Future Builder

[Peter Kriens]

There is a build on https://bndtools.ci.cloudbees.com/job/pkriens.newBuilder/lastSuccessfulBuild/artifact/build/generated/p2/ that I’d like to get some feedback on. It is a complete rewrite of the builder so it needs to be thoroughly tested. Key difference is that now the order is defined by bnd and not Eclipse. That is, all bnd projects are forced to be build in bnd order. (Non bnd projects are kept in the same order and are build after all bnd projects are build.) The Future Builder is more aggressively trying NOT to build so be aware.

If you put build logging on (bndtools Eclipse preferences) then you can see a complete decision tree in the Error log why a project is decided to be build or not.

My feeling is that large builds are significantly faster. However, before we can integrate this in the main build I need to get some feedback that it actually works for all existing workspaces.

Kind regards,

Peter Kriens

[Ferry Huberts]

On 11/02/15 16:00, Peter Kriens wrote:

There is a build on https://bndtools.ci.cloudbees.com/job/pkriens.newBuilder/lastSuccessfulBuild/artifact/build/generated/p2/ that I’d like to get some feedback on. It is a complete rewrite of the builder so it needs to be thoroughly tested. Key difference is that now the order is defined by bnd and not Eclipse. That is, all bnd projects are forced to be build in bnd order. (Non bnd projects are kept in the same order and are build after all bnd projects are build.) The Future Builder is more aggressively trying NOT to build so

As I've tried to explain: this is not good enough, see the relevant mail thread.

If you're not going to fix that, then I'd really like to see an option/setting (settable from build.bnd) that keeps the current build behaviour.

be aware.

If you put build logging on (bndtools Eclipse preferences) then you can see a complete decision tree in the Error log why a project is decided to be build or not.

My feeling is that large builds are significantly faster. However, before we can integrate this in the main build I need to get some feedback that it actually works for all existing workspaces.

Kind regards,

Peter Kriens

--
You received this message because you are subscribed to the Google Groups "bndtools-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
Ferry Huberts

[Peter Kriens]

On 11 feb. 2015, at 18:09, Dave <dhum...@gmail.com> wrote:

If you put build logging on (bndtools Eclipse preferences) then you can see a complete decision tree in the Error log why a project is decided to be build or not.

What does the log output mean exactly? If I make a code change say to xxx.integration project I see something like:

xxx.integration no build

xxx.integration 4 files were build

If I make a change the project but there is a compile error I see:

xxx.integration no build

If I make a change to the impl project I see:

xxx no build

xxx 1 file was build

I take this to mean:

* both no build and was/were build: it actually built the project

The reason that you see no build is that Eclipse calls the builder multiple times. First time there is a delta that triggers a build but then (since I change the generated directory) it calls me again with that change which I ignore. I do not think I can tell Eclipse to not call me for certain directories.

* no build (only): project is in the dependency chain, but did not try/could not complete the build

* <no message>: project not on dependency chain, did not build

I do not recognize those messages. Can you send them? Do you have non bnd projects?

The “no build" message is what is confusing.

I hope this is clear now :-)

BJ also has made some changes that look really good (and actually get rid of the global ordering, he found a way to specify the ordering per project). So watch this space.

Thanks for the feedback, kind regards,

Peter Kriens

Still need to do more testing to see if things are faster, but it seems like it would be.

Thanks,

Dave

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.

[Peter Kriens]

Good point, though -classpath is not recommended to be used in a Project (it was a feature to use the Builder standalone) it actually is a binary dependency. So I need to include these files in checking for what is changed. I.e. I ignore your enhance_bin folder right now. 

Sounds like a scary setup though. These classes have the same name as the classes in bin?

I would suggest to not use -classpath but instead add them to -includeresource

-includeresource: enhance_bin

This will then not only work appropriately in the build but it also guarantees overwriting anything from bin since resources are done copying from the classpath. The dependency checker is aware of what -includeresource is using.

Kind regards,

Peter Kriens

On 11 feb. 2015, at 18:34, Dave <dhum...@gmail.com> wrote:

I have Ant tasks that must be run to build some resources. At some point I would like to integrate those more so into bnd build, but until then with the older (current) build I can run the Ant task, refresh the project containing the Ant built resources and bnd will build the bundle with the newly built resources.

However, it seems like the "future" builder works differently. For example, I have some class files that are enhanced by DataNucleus through an Ant target. The enhanced class files go in an enhance-bin folder. When I refresh the project (so Eclipse knows about the updated resources), bnd does not build the bundle (I don't see any messages in the error log like no analysis is done). I do have "-classpath: enhance-bin" in the bnd file so the bin folder should be picked up. If I delete the bundle, a new bundle will be created with the enhanced class files. Maybe, this is just an event that needs to be handled?

Also, I notice the builder analyzes projects before any Ant target is run from Eclipse. I wouldn't think this would be necessary. Is there a reason for this?

Thanks,

Dave

[Peter Kriens]

Include-Resource === -includeresource no difference. I started with the Include-Resource but found out it was confusing to have this header in the manifest. It ended up there because it started with an Upper case. So then I created -includeresource as an alias.

Kind regards,

Peter Kriens

On 12 feb. 2015, at 14:11, Dave <dhum...@gmail.com> wrote:

We are actually using "Include-Resource" to include non Java class files for that bundle. How does that header compare with -includeresource? Are they basically the same thing? What happens if both are set?

[Ferry Huberts]

Ok, so I've looked into the commit that changes the build order.

I can't express enough how much I object to that commit: it _completely_ ignores non-bnd projects!!!
That immediately disqualifies bndtools for some of my workspaces, which to me is a MAJOR regression.

I'm also getting the distinct feeling that you don't like hearing this feedback, I've received no feedback to that effect.
I don't mind being wrong (or right) and being told so, but no feedback is rather 'disappointing'.

As said before:

On 06/02/15 19:46, BJ Hargrave wrote:

On Feb 6, 2015, at 13:37 , Ferry Huberts <mail...@hupie.com> wrote:

I can have a bnd project that depends on a non-bnd project

How do you express this in the bnd project's bnd.bnd file? In the -buildpath?

I don't have it personally, but you can express this in Eclipse's project settings.
You'd then also need to express this in the offline build ofcourse.

It's possible.
I'm not saying how, just that that it's possible.

And from what I remember is that your OSGi build has/had it too; cnf supposedly contained code that needs/needed to be built before other projects.

(and visa-versa)

This should be covered by putting the bnd projects earlier in the build order than non-bnd projects.

yes.

However, both situations can be present _at_the_same_time_.
There might even be projects that both depend on one or more bnd projects _and_ are dependencies for one or more bnd projects.

[..experiencing flashbacks to the Gradle situation of last year...]

[Peter Kriens]

If both are set, one is ignored. You an look in the source 

String includes = getProperty("Bundle-Includes");

if (includes == null) {

includes = mergeProperties(INCLUDERESOURCE);

if (includes == null || includes.length() == 0)

includes = mergeProperties(Constants.INCLUDE_RESOURCE);

} else

warning("Please use -includeresource instead of Bundle-Includes");

So if -includeresource is not set, bnd looks at Include-Resource. And as you can see, there even was a Bundle-Includes long ago … one day someone will write bnd’s history from these artifacts. :-)

Kind regards,

Peter Kriens

On 12 feb. 2015, at 14:11, Dave <dhum...@gmail.com> wrote:

We are actually using "Include-Resource" to include non Java class files for that bundle. How does that header compare with -includeresource? Are they basically the same thing? What happens if both are set?

On Thursday, February 12, 2015 at 3:48:28 AM UTC-5, Peter Kriens wrote:

[BJ Hargrave]

I have released my changes to a new feature branch on the bndtools repo so Peter and I can collaborate further: https://github.com/bndtools/bndtools/tree/futurebuilder

The jenkins build for this branch can be installed from the update site at: https://bndtools.ci.cloudbees.com/job/bndtools.futurebuilder/lastSuccessfulBuild/artifact/build/generated/p2/

As Peter noted, I made a change to the builder so that it no longer attempts to set the global build order but rather use's a bnd project's dependsOn list to set dynamic project references in Eclipse. This allows Eclipse to continue to use the default build order it computes but using the additional project reference information set by bndtools. See https://github.com/bndtools/bndtools/commit/18b1a7aeebfa071b290dacb3e3c1574efca6a769 for this change.

We still have other issues to address (currently the builder does a full workspace build each time you start Eclipse), but this change is looking very good.

You received this message because you are subscribed to the Google Groups "bndtools-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

-- 

BJ

[BJ Hargrave]

Peter and I have completed the work and have integrated the new builder and new classpath container implementation (thanks Carter!) into the master branch. So the futurebuilder branch and build are now obsolete. 

Please try out the latest bndtools 3.0 builds from https://bndtools.ci.cloudbees.com/job/bndtools.master/lastSuccessfulBuild/artifact/build/generated/p2/ and give us feedback.

-- 

BJ

[Ferry Huberts]

On 16/02/15 23:18, BJ Hargrave wrote:

Peter and I have completed the work and have integrated the new builder and new classpath container implementation (thanks Carter!) into the master branch. So the futurebuilder branch and build are now obsolete. 

Does this address my concerns about non-bnd projects?
And in what way?
Are non-bnd project correctly placed in the build order, even when they depend on bnd projects _and_ they are dependencies for bnd projects?

Nobody has bothered to reply to any of my emails about that so it's 'rather difficult' to get of sense of what is going on.


And what's the deal with the 'packages=**' attirubute on some build dependencies? What does it do and why is it needed?

-- 
Ferry Huberts

[Peter Kriens]

I’ve been using this build now for a couple of days and it is a major improvement over what we had. If you had large builds with lots of dependencies then you will love this :-) We combined the work of Carter with this build which significantly improves the access rules settings. You will likely get quite a few warnings about restricted access but these are all correct indications that you’re doing something you maybe shouldn’t.

Major changes:

• Access violations (e.g. using private, or more accurate not exported, packages) are now warnings
• We now faithfully use the build JAR file instead of the bin folder with version=latest. Version=project uses the bin folder directly. Using the JAR means that any extra packaged code will be visible in downstream projects.
• F3 et al work despite having a binary reference
• Project dependencies are now faithfully recorded in the project’s dynamic dependencies, this results in building the projects in proper bnd order.
• You can add a ‘packages’ attribute on a bundle in the -buildpath with a (comma separated) wildcarded expression for packages that are not exported but should not generate access warnings.
• Non OSGi bundles are fully visible, they generate no warnings
• cnf will refresh if any of the build.bnd, ext/*.bnd, or included files have changed
• Project is now rebuild when:
• cnf was refreshed
• Any resource in the project that is not in generated, src, test, bin_test has changed
• Any upstream dependency has changed its generated/buildfiles file
• Error markers in test are not blocking the build. In src they are blocking by default but this can be changed in the preferences.
• The option to include Eclipse build path dependencies is removed
• Code is now split in separate classes for handling markers & errors, deltas, cnf, and general build making it a lot more easy to maintain.
• Workspace is refreshed if you actually build JARs in the cnf project (yes you can …) so that any plugins that are build in the cnf project are refreshed. 

Open issues:

• We probably need to watch upstream dependency’s bin folder if version=project. In practice, this generates a new compile for downstream projects, which should trigger the bnd build. However, there could be a possibility we do not pick up changes. Needs some further thought.
• BJ and I discussed having separate access rules for src & test. This seems impossible with the current Eclipse model but would be really nice. Would be nice to hear about someone that has knowledge of this stuff inside Eclipse.
• There are now even more duplicates in Open Type because we get them through the JAR and the bin folder. The two deps are there to handle Eclipse navigation & the dep on the JAR. Would be nice if we could filter those (or at least sort) so that the source project is on top.

It looks awfully good but please realize this is in the ’under construction’ build. No guarantees so no whining accepted, but positively formatted feedback is of course always more than welcome.

Kind regards,

Peter Kriens

[Ferry Huberts]

I've tried it a bit, not extensive by far, works so far.

One quite annoying thing is that code completion no longer shows the parameter names but generic arg0, arg1, etc names.
That - to me - is loss of usability/friendlyness.

-- 
Ferry Huberts

[Ferry Huberts]

On 18/02/15 10:46, Ferry Huberts wrote:

I've tried it a bit, not extensive by far, works so far.

One quite annoying thing is that code completion no longer shows the parameter names but generic arg0, arg1, etc names.
That - to me - is loss of usability/friendlyness.

I think that can be fixed by letting 'version=latest' be same as 'version=project' again, like it was before.
These are equivalent anyway.

-- 
Ferry Huberts

[Timothy Ward]

Hi Ferry,

That sounds like a lack of proper source attachment - do you see the same thing happening with version=project and version=latest? Also, are the affected projects including the source in the Jar file, and if not does that make a difference? 

Eclipse Classpath containers can be a real pain!

Tim 

[Ferry Huberts]

On 18/02/15 10:51, Timothy Ward wrote:

Hi Ferry,

That sounds like a lack of proper source attachment - do you see the same thing happening with version=project and version=latest? Also, are the affected projects including the source in the Jar file, and if not does that make a difference? 

Ah so it appears I was wrong since the generic arg names are always shown, no matter if I use version=latest or version=project.
Sources are in the bundles.

And I've also just confirmed that before the 'new builder' it worked just fine.

-- 
Ferry Huberts