Discussion: New Custom Jenkins WAR packaging tool

[Oleg Nenashev]

Hi all,

Currently I am working on setting up testing flows for my prototypes (e.g. JENKINS-38313 - External Build Logging or JENKINS-44100 - Detach Remoting logic from the core). Such prototypes include multiple components including unreleased core patches and plugins. Our current test frameworks (Jenkins Test Harness, Acceptance Test Harness, Plugin Compatibility Tester) are mostly focused on WAR file testing, so I would need to package the prototypes as WAR files in order to run tests.

In order to achieve the goal, I have created a new packaging tool: https://github.com/oleg-nenashev/jenkins-war-packager, which is generally a maven-hpi-plugin:custom-war on steroids. It can run as CLI tool and as a Maven plugin. The tool takes the WAR as a YAML definition (example) and packages everything as a single WAR file:

• Base WAR (including dev branches from Git/filesystem)
• List of plugins and libraries to be bundled (including dev branches from Git/filesystem)
• List of system properties to be pre-set
  
• Groovy Hook Scripts to bundle (may also support Configuration-as-Code plugin later)
  

WAR file includes system properties and Groovy hooks, so it may configure itself on startup. It allows the custom configuration to be transparently used with existing tests.

I have created two examples for the tool:

• External Task Logging for ElasticSearch - A prototype we created together with Jesse Glick in 2016. It packages extra plugins and preconfigures them
  
• Jenkins - all latest WAR - Automatic build of the Jenkins core with latest dependencies (Stapler, Remoting, modules)
  

In the current state I consider Custom Jenkins War Packager to b e a development tool only. It can be potentially used to package Jenkins WARs for production use, but IMHO it would require more work before it can be advertised (standardized specification format, support of Docker packaging, etc.). But Jenkins developers may find the tool useful even in the current state. So I would like to get it hosted within Jenkins organization and released as an alpha version.

If you are interested in such packaging and development tool for Jenkins, I would appreciate any feedback.

Best regards,

Oleg Nenashev

[Jesse Glick]

On Thu, Mar 22, 2018 at 8:25 AM, Oleg Nenashev <o.v.ne...@gmail.com> wrote:
> It allows the custom configuration to be transparently
> used with existing tests.

How would that work with JTH tests as run via PCT, since these will
only pick up a Jenkins WAR from the local Maven repository? Do you
need to `mvn install:install-file` the custom WAR as a `SNAPSHOT`
version that can be fed to PCT?

[Oleg Nenashev]

How would that work with JTH tests as run via PCT, since these will only pick up a Jenkins WAR from the local Maven repository

 
Actually they support other sources:

• JTH can take WAR from different sources, not only from local Maven repository (see WAR Exploder). I was a bit lazy during prototyping, so I was just putting WAR to war/target/jenkins
• PCT supports "-war" option, which allows specifying a custom WAR path. I was using this a lot during JEP-200 testing in December/January. The new PCT Docker image also supports custom WAR specification
  

Next week I will create a demo which actually runs JTH and PCT with a custom WAR file within the repository.

BR, Oleg

[Jesse Glick]

On Thu, Mar 22, 2018 at 2:58 PM, Oleg Nenashev <o.v.ne...@gmail.com> wrote:
>> How would that work with JTH tests as run via PCT, since these will only
>> pick up a Jenkins WAR from the local Maven repository
>

> JTH can take WAR from different sources, not only from local Maven
> repository (see WAR Exploder). I was a bit lazy during prototyping, so I was
> just putting WAR to war/target/jenkins

Uh…but that is only supposed to be used from `jenkinsci/jenkins`. We
should not abuse that for other purposes. We could introduce a new
system property or environment variable pointing to a custom WAR file
path for `WarExploder` to use, perhaps.

> PCT supports "-war" option, which allows specifying a custom WAR path.

Yes it does, but that has no direct effect on the file which is loaded
by `WarExploder`.

https://github.com/jenkinsci/plugin-compat-tester/blob/4898b916f81f813d202721e1fe0eb36d222977a5/plugins-compat-tester/src/main/java/org/jenkins/tools/test/PluginCompatTester.java#L155-L207

It is only used to pick up metadata about the core version to test
against, the list of plugins and their versions to test, and some
miscellaneous stuff like that. The actual test runs will go back to
the local Maven repository for a `jenkins-war` artifact with the
specified version number, unless `WarExploder` is patched as discussed
above.

> Next week I will create a demo which actually runs JTH and PCT with a custom WAR file within the repository.

Yes please—somehow verifying that the tests run against the actual
core and plugin versions specified in the YAML definition, even if
those are unreleased, present only in local source checkouts, etc.;
and that all plugins listed in the YAML are loaded during the test
execution, even if they are not present as `test`-scoped dependencies
in the POM of the plugin defining the test.

Also would be useful to see a demo of ATH tests running against this
WAR, with the same version constraints, and analogously verifying that
all listed plugins are loaded even if they are not mentioned in a
`@WithPlugins` annotation.

[Oleg Nenashev]

We could introduce a new system property or environment variable pointing to a custom WAR file
path for `WarExploder` to use, perhaps.

Yes, I had something like that in my TODO list for the final impl.
It would require plugins to update to the recent JTH, but it's not a problem since we can always pass Maven property when running builds in CI/PCT. It may fail in some cases (upper bounds, new injected tests, etc.), but should work for the most of the plugins

Yes it does, but that has no direct effect on the file which is loaded by `WarExploder`. The actual test runs will go back to the local Maven repository for a `jenkins-war` artifact with the specified version number

It is fine with me, because Custom War Packager installs artifacts to the local repo anyway.
Obviously there are some limitations on groupId and artifactId to make such approach running, but it works.

BR, Oleg

[Jesse Glick]

On Thu, Mar 22, 2018 at 8:19 PM, Oleg Nenashev <o.v.ne...@gmail.com> wrote:
>> We could introduce a new system property or environment variable pointing
>> to a custom WAR file
>> path for `WarExploder` to use, perhaps.
>

> It would require plugins to update to the recent JTH, but it's not a problem
> since we can always pass Maven property when running builds in CI/PCT.

I think it is fine to say that if you want this to work from your
plugin’s tests, it just needs to have a sufficiently new POM. That is
something we can require for Essentials plugins at least.

>> The actual test runs will go back to the local Maven
>> repository for a `jenkins-war` artifact with the specified version number
>
> It is fine with me, because Custom War Packager installs artifacts to the
> local repo anyway.
> Obviously there are some limitations on groupId and artifactId to make such
> approach running, but it works.

Good enough! Just please use `SNAPSHOT`-labelled versions so cron jobs like

find -L ~/.m2/repository \( -type d -name '*-SNAPSHOT' -prune -o -type
f -name maven-metadata-local.xml \) -exec rm -rf {} \;

can clean up.

[Oleg Nenashev]

I have published the current state of the project as 0.1-alpha-2. So Maven flow can be also used now. Since it is an alpha version, there may be incompatible changes later. But any feedback will be appreciated. If you see any defects or need any changes, please submit tickets in the GitHub issue tracker. I will move to JIRA once it reaches a stable state.

So far I am going to proceed with the better demo for CI flows with Custom WAR Packager.

Best regards,
Oleg

[Oleg Nenashev]

All of them would need to locally build the war file or share the local maven repo or something similar...

There is an ongoing discussion about a staging repo in IEP-9. I could add an option to Custom WAR packager to deploy artifacts to such staging repo.

In my CI instances I use maven-repo as a volume anyway to save the build time.

So I would say that adding the ability to set the location of the file to be loaded by `WarExploder` is going to be a real need if we want this to be in production 

Work in progress.

BR, Oleg

On Mon, Apr 2, 2018 at 10:29 AM, Raul Arabaolaza <rarab...@cloudbees.com> wrote:

Sorry for being late to this...

I agree with the proposed approach, the only thing that concerns me is that it seems you need to build the custom war locally so is installed on your local repo, now what happens with CI jobs that execute the PCT in parallel? All of them would need to locally build the war file or share the local maven repo or something similar...

So I would say that adding the ability to set the location of the file to be loaded by `WarExploder` is going to be a real need if we want this to be in production 

Regards, Raul

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/ySjSak-m8u0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/21c59a4e-9f92-4b27-934a-fbbb5e396bdb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.