How do I keep track of what was built together once published to a repository?

[t...@quarendon.net]

Currently we have things set up so that we use a maven repository as our release repository. 

However, the upshot is that I lose track of what was built, and tested, together.

Currently all our package and version numbers are "1.0", something that I'm, obviously 

keen to change :-)

So I intend to use api baselining, which as far as I understand it will magically check

that the package version numbers are suitable, and, I assume, automatically keep the 

bundle version numbers up to date too. 

From a "distribution" point of view, it's the bundle version numbers that are interesting. 

So on the face of it I would want to record the set of bundles that have been built by 

a particular build in the form of a kind of manifest of bundle symbolic names and version 

numbers that I can then also publish into the repository. I'm concerned that if I don't

do this automatically, I'll end up with a very error prone process by which I keep something

up to date manually and end up distributing to customers a set of artifacts that haven't

actually been tested together.

Why am I reconsidering maven? Why am I thinking this is relevant?

A number of comments I've read recently would seem to suggest that if you publish

"locally" to an OBR, you can then somehow deploy from that, you've captured 

the "manifest" if you like (sorry, I realise manifest is a poor choice of word in this context).

But that would suggest a repository *per build*, but then how do I "distribute" that?

Probably my question comes from a position of ignorance and misunderstanding, 

so please tell me what the "right" question is!

I'm just trying to work out how I distribute the result of my build. I'm assuming that

the result of one gradle build will constitute one "feature" (to use Karaf terminology)

but that my final distribution will ultimately consist of pulling together multiple of these

"features" from different builds from source in different git repositories into one

distribution, ultimately to be hosted in a Karaf container.

Apologies if my questions are not clear. I think I might have caused

my brain to overheat trying to understand how to manage versions and the definition

of a "product". So any experiences would be great.

Thanks.

[Raymond Auge]

On Fri, Jun 16, 2017 at 9:52 AM, <t...@quarendon.net> wrote:

Currently we have things set up so that we use a maven repository as our release repository. 

However, the upshot is that I lose track of what was built, and tested, together.

Currently all our package and version numbers are "1.0", something that I'm, obviously 

keen to change :-)

So I intend to use api baselining, which as far as I understand it will magically check

that the package version numbers are suitable, and, I assume, automatically keep the 

bundle version numbers up to date too. 

From a "distribution" point of view, it's the bundle version numbers that are interesting. 

This isn't exactly true and it's what the OSGi experts have been trying to express to everyone for a very long time.

Rather, because every package (and other cap&req) in every bundle to be contained in a "distribution" is semantically versioned, you can use the resolver to calculate the set of "distribution" bundles.

Doing so ensures that, with pretty much absolute certainty, a generated system can:
1) run, since all requirements are satisfied
2) never fail on something like a CNFE or on any missing things

Using the resolver for calculating the set of "distribution" bundles is pretty easy nowadays.

* In  maven it's using the bnd-resolver-maven-plugin:

https://github.com/bndtools/bnd/tree/master/maven/bnd-resolver-maven-plugin

* In (pure) gradle it's using the aQute.bnd.gradle.Resolve task:

https://github.com/bndtools/bnd/blob/master/biz.aQute.bnd.gradle/README.md#create-a-task-of-the-resolve-type

* In bndtools gradle it's using the built in `resolve` operation:

https://github.com/bndtools/bnd/blob/master/biz.aQute.bnd.gradle/README.md#additional-tasks

Sincerely,

- Ray

 

So on the face of it I would want to record the set of bundles that have been built by 

a particular build in the form of a kind of manifest of bundle symbolic names and version 

numbers that I can then also publish into the repository. I'm concerned that if I don't

do this automatically, I'll end up with a very error prone process by which I keep something

up to date manually and end up distributing to customers a set of artifacts that haven't

actually been tested together.

Why am I reconsidering maven? Why am I thinking this is relevant?

A number of comments I've read recently would seem to suggest that if you publish

"locally" to an OBR, you can then somehow deploy from that, you've captured 

the "manifest" if you like (sorry, I realise manifest is a poor choice of word in this context).

But that would suggest a repository *per build*, but then how do I "distribute" that?

Probably my question comes from a position of ignorance and misunderstanding, 

so please tell me what the "right" question is!

I'm just trying to work out how I distribute the result of my build. I'm assuming that

the result of one gradle build will constitute one "feature" (to use Karaf terminology)

but that my final distribution will ultimately consist of pulling together multiple of these

"features" from different builds from source in different git repositories into one

distribution, ultimately to be hosted in a Karaf container.

Apologies if my questions are not clear. I think I might have caused

my brain to overheat trying to understand how to manage versions and the definition

of a "product". So any experiences would be great.

Thanks.

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Raymond Augé (@rotty3000)

Senior Software Architect Liferay, Inc. (@Liferay)

Board Member & EEG Co-Chair, OSGi Alliance (@OSGiAlliance)

[Raymond Auge]

Also note that all three of those build scenarios also have a simple steps to dump the set of bundles calculated by the resolver for you into something like either just a directory (for use in building other deployment target archives like karaf features), or a completely self contained executable jar.

Sincerely,

- Ray

[t...@quarendon.net]

On Friday, 16 June 2017 15:16:45 UTC+1, Raymond Auge wrote:

Also note that all three of those build scenarios also have a simple steps to dump the set of bundles calculated by the resolver for you into something like either just a directory (for use in building other deployment target archives like karaf features), or a completely self contained executable jar.

The "gradle export" produces a single executable jar based on a bndrun file. This doesn't seem useful for deployment into a container, as it's a self contained jar. This doesn't feel like the right thing to use for distributing a product, and anyway, doesn't allow me to combine it with artifacts from other builds.

Using "gradle runbundles" produces a directory with *all* the bundles in, including all of the external dependencies, which isn't really what I want. I would have to process that list, filter by name on all bundles I'm interested in, parse out the bundle symbolic name and the version in order to produce a karaf feature file. I guess I can do that, but it seems like I shouldn't have to.

Both of those processes start from a list of bundle names. I don't start with a set of package names. My bndrun file lists bundles. Physically, things are packages as jars which are named using the bundle symbolic name. If I want to deploy something, I have to physically copy a bundle jar by name, or reference it as a maven artifact etc. This is what I mean by the the bundle symbolic names and version numbers being interesting from a distribution point of view. I understand how the resolution works, in terms of capabilities and so on, and how an OBR is indexed by capability etc. 

In order to deploy something I need a list of file names, or bundle names. I don't understand how I would deploy something without doing that. Either I need to physically copy generated bundles somewhere, or I need refer to them as maven artifacts or something. All of those things are done using the bundle symbolic name. 

Sorry if I'm being a bit dim, but you might have to help me out with some specifics on how I would, say, take the output of gradle runbundles and "deploy" the result into, say, karaf. I can easily get the results published into a maven bundle, but I loose track of what is a coherent set of bundles.

I'm clearly not expressing my problem in a way that's understandable. What I *think* I want to do is create a karaf feature.xml file off the back of the gradle build, that I can then use to deploy what I've just built. But it should only contain the bundles under development. I'm happy to deal with the prereqs "manually" as it where, and trust that the resolution process that occurs when you deploy the feature into karaf will ensure that all the requirements have been met. 

Currently that's the only way I can understand of going from a source tree to a container containing running code. If I shouldn't be doing that and should be doing something else instead, please help me to understand!

Thanks

[BJ Hargrave]

In the Bnd build, we release to a "remote" maven repo which is just on the local file system (dist/bundles). So, at the end of the build, you can see all the released artifacts to know what you built and released. You can release to multiple repos at the same time (-releaserepo: A, B), so you can release to a local file system repo and a real nexus remote repo if you need.

--

You received this message because you are subscribed to the Google Groups "bndtools-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BJ

[Raymond Auge]

On Fri, Jun 16, 2017 at 11:39 AM, <t...@quarendon.net> wrote:

On Friday, 16 June 2017 15:16:45 UTC+1, Raymond Auge wrote:

Also note that all three of those build scenarios also have a simple steps to dump the set of bundles calculated by the resolver for you into something like either just a directory (for use in building other deployment target archives like karaf features), or a completely self contained executable jar.

The "gradle export" produces a single executable jar based on a bndrun file. This doesn't seem useful for deployment into a container, as it's a self contained jar. This doesn't feel like the right thing to use for distributing a product, and anyway, doesn't allow me to combine it with artifacts from other builds.

Using "gradle runbundles" produces a directory with *all* the bundles in, including all of the external dependencies, which isn't really what I want.

Actually that's exactly what the "distro" feature is for... it helps you calculate the set of bundles only needed to deploy against your distro.

Creating your distro is very simple. It's kind of like stating the version of karaf, except it's an index jar which you can share by placing it in source control, or in maven (i'll just demonstrate source control example.)

1 ) create the distro (jar) for your target "deployment" by installing this bundle [1]

2) using this jar [2], execute the command:

java -jar <bnd.jar> remote distro my.karaf.distro <version.of.karaf> -o my.karaf.distro-<version.of.karaf>.jar

3) in your bndrun file use this setting:

-distro: ${.}/my.karaf.distro-<version.of.karaf>.jar;version=file

4) now when you run the resolver and then `runbundles` the directory will contain ONLY the bundles necessary to deploy and NONE of the things provided by the distro.

5) Finally, you can make your own task which then takes those bundles and builds an actual feature for you.

Note I'm suggesting using the latest bnd RC build because bnd developers worked explicitly on this process during this last development iteration. However, you shouldn't need to change bnd build plugins used for the build (but it's also a good idea to use the RC at this point).

[1] https://bndtools.ci.cloudbees.com/job/bnd.next/599/artifact/dist/bundles/biz/aQute/bnd/biz.aQute.remote.agent/3.4.0-SNAPSHOT/biz.aQute.remote.agent-3.4.0-20170605.151139-1.jar
[2] https://bndtools.ci.cloudbees.com/job/bnd.next/599/artifact/dist/bundles/biz/aQute/bnd/biz.aQute.bnd/3.4.0-SNAPSHOT/biz.aQute.bnd-3.4.0-20170605.151041-1.jar

Sincerely,

- Ray

 

I would have to process that list, filter by name on all bundles I'm interested in, parse out the bundle symbolic name and the version in order to produce a karaf feature file. I guess I can do that, but it seems like I shouldn't have to.

Both of those processes start from a list of bundle names. I don't start with a set of package names. My bndrun file lists bundles. Physically, things are packages as jars which are named using the bundle symbolic name. If I want to deploy something, I have to physically copy a bundle jar by name, or reference it as a maven artifact etc. This is what I mean by the the bundle symbolic names and version numbers being interesting from a distribution point of view. I understand how the resolution works, in terms of capabilities and so on, and how an OBR is indexed by capability etc. 

In order to deploy something I need a list of file names, or bundle names. I don't understand how I would deploy something without doing that. Either I need to physically copy generated bundles somewhere, or I need refer to them as maven artifacts or something. All of those things are done using the bundle symbolic name. 

Sorry if I'm being a bit dim, but you might have to help me out with some specifics on how I would, say, take the output of gradle runbundles and "deploy" the result into, say, karaf. I can easily get the results published into a maven bundle, but I loose track of what is a coherent set of bundles.

I'm clearly not expressing my problem in a way that's understandable. What I *think* I want to do is create a karaf feature.xml file off the back of the gradle build, that I can then use to deploy what I've just built. But it should only contain the bundles under development. I'm happy to deal with the prereqs "manually" as it where, and trust that the resolution process that occurs when you deploy the feature into karaf will ensure that all the requirements have been met. 

Currently that's the only way I can understand of going from a source tree to a container containing running code. If I shouldn't be doing that and should be doing something else instead, please help me to understand!

Thanks

--

You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[t...@quarendon.net]

Actually that's exactly what the "distro" feature is for... it helps you calculate the set of bundles only needed to deploy against your distro.

I had seen reference to the "-distro" option, but hadn't investigated it. I'll have to have a play around with that.

It sounds like the result will be that I can more easily come up with the list of bundles that I need to deploy, including any additional third party dependencies. 

I can see I can then probably create a features file containing two features, one containing "our" bundles, and one containing "third party" bundles, simply filtering by name.

Thinking about it though I'm not sure how this would scale to another build where the "third party dependencies" (i.e bundles that are not the "bundles under development") are *also* our bundles.

If you see what I mean.

Anyway, you've given me some ideas to think about.

Thanks.

[t...@quarendon.net]

On Friday, 16 June 2017 16:55:45 UTC+1, BJ Hargrave wrote:

In the Bnd build, we release to a "remote" maven repo which is just on the local file system (dist/bundles). So, at the end of the build, you can see all the released artifacts to know what you built and released. You can release to multiple repos at the same time (-releaserepo: A, B), so you can release to a local file system repo and a real nexus remote repo if you need.

Ah, OK. That sounds useful. I'll have a look into that. It might then be easier to then process the index.xml file from the repository to pull put the list of bundles and versions, and from that generate the "manifest". 

I'll experiment with that. It sounds promising.

Thanks.