Determining release artifacts before build

[David Leangen]

Hi!

Another question about bnd/gradle. 😬


Now that I have included npm in the mix, build time has (predictably) increased a **LOT**. I want to improve my CI build time. One thought is to only build those projects that will actually be released.

Whether or not a bundle is released is determined by baselining. Only those projects that have a newer version number get released. Those with an unchanged version number do not get released. It is therefore a bit of a waste of resources to build projects that have not changed. I would like to be able to skip the build for such project when there is no need to build it.


So, I would like to somehow build only those projects that (1) are on a list of projects that should always be built regardless, and (2) projects that should only be built if the version has changed.

The ultimate goal of course is for the export task to use these built bundles along with all other bundles that are already available in my repo.


I tried looking here for some hints:

https://github.com/bndtools/bnd/blob/master/biz.aQute.bnd.gradle/src/aQute/bnd/gradle/BndPlugin.groovy#L364-L368
https://github.com/bndtools/bnd/blob/master/biz.aQute.bnd.gradle/src/aQute/bnd/gradle/BndBuilderPlugin.groovy
https://github.com/bndtools/bnd/blob/master/biz.aQute.bnd.gradle/src/aQute/bnd/gradle/BndPlugin.groovy#L346
https://github.com/bndtools/bnd/blob/master/biz.aQute.bnd.gradle/src/aQute/bnd/gradle/Baseline.groovy

Then I gave up and thought it would be much quicker to just ask the experts. 😅

Is there a simple way of accomplishing this already without writing a custom plugin?


Thanks!
=David

[Peter Kriens]

You're sure the increased complexity and your work is worth this effort? I've actually no idea how much commercial builds costs on Travis/Github but since they can make it free for open source it sounds you need to a LOT of building to save one of your workdays?

I would keep it simple until it hurts.

PK

> --
> You received this message because you are subscribed to the Google Groups "bndtools-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/DBC56D59-65FE-4433-8290-8B682BADE020%40gmail.com.

[David Leangen]

Thanks Peter.

> it sounds you need to a LOT of building to save one of your workdays?

I think that indirectly this answers the original question:

>> Is there a simple way of accomplishing this already without writing a custom plugin?

I take it the answer is “no”. 😀

In that case I completely agree with this:

> You're sure the increased complexity and your work is worth this effort?

Yes, I am absolutely sure that it is NOT worth the effort.


Thanks!
=David

[Jürgen Albert]

Hi David,

First of all: I totally agree with Peter. You don't want to go down this
rabbit hole! The effort would simply be unreasonable.

Long build times however can be bothersome. One needs to decide how
much. If the build server takes a long time, I believe this is okay.
This it, what a System CI is for: to do the jobs that need doing. Long
build times for the developer in the IDE on the other hand can be a real
pain in the ass and decrease the efficiency of a developer.

Some things can't be helped, like unit tests that take a long time or
NPM. Sometimes it simply is what it is. Besides that, we found that long
build times can be a sign off an unnecessary complex structure. If we
have such a Situation we usually ask ourself the following questions:

* Does everything need to be build together? Meaning do we have bundles
that have a low change frequency, that can be put in a separate
Workspace/Repo (like e.g. Framework, Core or Common).
* Do we have bundles that change frequently and cause a lot of dependent
projects to be build? Typical candidates are e.g. common bundles where
you keep your Util classes. Do all classes need to be in one bundle or
can they become separate bundles or part of the functional bundle where
they belong?

To conclude: Long build times can be a sign of too much complexity. The
build simply exposes this. Improve the build speed by restructuring
usually leads automatically to an improvement in your overall architecture.

I hope this helps,

Jürgen.

--
Jürgen Albert
Geschäftsführer

Data In Motion Consulting GmbH

Kahlaische Str. 4
07745 Jena

Mobil: 0157-72521634
E-Mail: j.al...@datainmotion.de
Web: www.datainmotion.de

XING: https://www.xing.com/profile/Juergen_Albert5

Rechtliches

Jena HBR 513025

[David Leangen]

Hi Jürgen,

> To conclude: Long build times can be a sign of too much complexity. The build simply exposes this. Improve the build speed by restructuring usually leads automatically to an improvement in your overall architecture.

Thanks. That is good advice.

In this case, the only difference from before was adding npm. Just this change made the build “unreasonable”. The most wasteful part is that the npm plugin runs `npm install` for each web project, which is totally wasteful. I am investigating if there is a way to use a shared `/node_modules/` folder.

But aside from that, you make a good point about using a practical structure. Up until now, I have been using a very puristic structure for my workspaces based only on the semantics of the system, not on the practicalities. It actually didn’t even enter my mind to think about the problem that way.

You have given me food for thought.


Cheers,
=David

[Peter Kriens]

Yeah, I am using yarn at a customer and it is surprisingly heavy to build a single page web app :-( I can see the value it (tries) to provide but I am not convinced it is worth the effort. Nowadays browsers are pretty standardized and with a few rules you can write a single page web app in Javascript & HTML 5. I am suspicious that much of the stuff that yarn et al are doing is only there to be able to use exotic languages nowadays and not having to think about where your assets are. I also noticed that nowadays you cannot even develop inside chrome anymore since it refuses to server things from the file systems due to security reasons.

Then again, I've not written a single page web app for quite some time. But what I see is that it started out with the goal of doing it all very much simpler than Java but ended being more complex.

Good luck.

Kind regards,

Peter Kriens

> --
> You received this message because you are subscribed to the Google Groups "bndtools-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/03B2C0CD-E906-49B9-AD15-25ABB9867346%40gmail.com.

[BJ Hargrave]

The problem is that the only way you can know of a bundle has changed (via baselining) is to build the bundle first. At that point, you have not saved yourself from building the bundle. There is no way to baseline a bundle without having built the bundle as the bundle is built from many, many inputs.

The best solution for you is to make sure your gradle task inputs and outputs are high quality and correct which lets gradle decide which tasks need to be executed. This can help a lot for incremental builds on a developer's workstation. But not so much for a CI build which should always build everything to ensure a reliable and predictable build. I am also not a fan of CI build caching (like caching .m2/repository across CI build runs) as this itself can be a source of build problems sometimes.

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/DBC56D59-65FE-4433-8290-8B682BADE020%40gmail.com.

--

BJ

[David Leangen]

Thank you for this great information. It is crystal clear now that this is not the way forward.

The build in its current form, using npm, is neither practical nor scalable. It is not practical (at least compared to before) because now when I want to deploy I can no longer do it in one sitting. It is not scalable because it wouldn't take a huge number of npm projects to make the build time so long that it becomes a huge bottleneck. I think I will need to go back to what I was doing before.

Meaning: I run npm run build during the development, generate the distribution artifacts, then deploy. The CI server will use these artifacts to create my bundles and deploy the final exported app.

Later I just may try out something fancy with branches, as Jürgen suggested. I can imagine for instance having a magic word for a branch name that would deploy a single artifact, only when needed, without having to rebuild everything for no good reason. Then the master build would continue to deploy the exported app, based on the previously deployed artifacts.