Setup external Jenkins processes to test via JCasC

[Oliver Gondža]

The other day, I have spent a week on crafting a reusable JUnit rule to
complement JenkinsConfiguredWithCodeRule[2] that create Jenkins SUTs as
independent local processes. The use case here is starting Jenkins with
realistic classloading environment, testing realistic start/stop/restart
scenarios (otherwise affected by JUT) or setting up multiple Jenkinses
for test. I am distributing this to see if folks are interested in
having something like this available so we can extract it from the
plugin this was initially developed for.

---

The simplest way to consume it is:

```
public @Rule TemporaryFolder tmp = new TemporaryFolder();
public @Rule ExternalJenkinsRule ejr = new ExternalJenkinsRule(tmp);

@Test
@ExternalFixture(name = "my-jenkins", resource = "my-jenkins.yaml",
injectPlugins = "matrix-auth")
public void myJenkins() throws Exception {
ExternalJenkinsRule.Fixture myJenkins = ejr.fixture("my-jenkins");
com.offbytwo.jenkins.JenkinsServer jenkinsServer =
myJenkins.getClient();
...
}
```

- The Jenkinses are launched in parallel in case there are multiple of them.
- Injected plugins are inject with their dependencies resolved by
maven-hpi-plugin:resolve-test-dependencies (IOW, they have to be
declared as maven dependencies but their injections is safe and fast).
- Primary means of controlling such Jenkins is through
jenkinsci/java-client-api.
- Dedicated temporary folder is allocated for every Jenkins.
- Jenkins log can be accessed for investigation.

For advanced usage one can subclass the ExternalJenkinsRule (as
demonstrated by [1]) in order to:

- Rearrange or modify the declared fixture annotations per test (DRY).
- Alter plugins to inject (DRY).
- Specify particular environment variables, JVM or Jenkins arguments.
- All of the above can be tailored based on declared fixture role.
Meaning the customizations deployed to individual fixtures do not have
to be the same - common for heterogeneous grids.

There are certainly many areas to improve this, though for now my
question is: Does it sound useful to you?

[1] https://github.com/jenkinsci/node-sharing-plugin/pull/106
[2]
https://github.com/jenkinsci/configuration-as-code-plugin/blob/master/plugin/src/test/java/io/jenkins/plugins/casc/misc/JenkinsConfiguredWithCodeRule.java

Cheers
--
oliver

[Jesse Glick]

On Thu, Feb 14, 2019 at 6:34 AM Oliver Gondža <ogo...@gmail.com> wrote:
> Does it sound useful to you?

Yes!

My main feedback is that, if possible, this should be reworked to use
a Docker container to launch Jenkins, with the image configurable on a
per-test basis. (That does not preclude mounting Maven-specific files
like `jenkins.war` and dependency `*.jpi` from the local repository or
`target` directory.) This would allow for much better control over
tests. I have sorely missed this in `mercurial-plugin`, for example.
There probably also needs to be some work done to allow agents to also
run in containers, like `docker-fixtures` does, with ports bound.
(Docker Compose? Minikube / microk8s? Something based on
testcontainers.org?)

I suggest you assign

https://issues.jenkins-ci.org/browse/JENKINS-41827

to yourself, as you seem to be implementing something very much like
what this wished for.

[Oliver Gondža]

On 14/02/2019 16.08, Jesse Glick wrote:
> On Thu, Feb 14, 2019 at 6:34 AM Oliver Gondža <ogo...@gmail.com> wrote:
>> Does it sound useful to you?
>
> Yes!
>
> My main feedback is that, if possible, this should be reworked to use
> a Docker container to launch Jenkins, with the image configurable on a
> per-test basis. (That does not preclude mounting Maven-specific files
> like `jenkins.war` and dependency `*.jpi` from the local repository or
> `target` directory.) This would allow for much better control over
> tests. I have sorely missed this in `mercurial-plugin`, for example.
> There probably also needs to be some work done to allow agents to also
> run in containers, like `docker-fixtures` does, with ports bound.
> (Docker Compose? Minikube / microk8s? Something based on
> testcontainers.org?)

Can you be more specific on the use-case here? As I understand, you
suggest to use container as a per-test fixture (ala docker-fixtures) and
running Jenkins _inside_ of the container.

--
oliver

[Jesse Glick]

On Mon, Feb 18, 2019 at 4:08 AM Oliver Gondža <ogo...@gmail.com> wrote:
> Can you be more specific on the use-case here? As I understand, you
> suggest to use container as a per-test fixture (ala docker-fixtures) and
> running Jenkins _inside_ of the container.

Yes. I think there are potentially a lot of use cases here, but to
take one example from my personal experience with the `mercurial`
plugin: while I can and do use `docker-fixtures` to run tests in which
an _agent_ runs in a specific image (where particular versions of
Python and Mercurial are installed), the situation is currently
hopeless when the test is exercising code whereby the _master_ needs
to run `hg` commands (for example: form validation or branch
indexing). These tests are currently written to just use whatever copy
of Mercurial happens to be installed on the computer running `mvn
test`, or to mark the test skipped if there is none (very likely in
CI).

Besides making external tools / commands available to the Jenkins
process, a harness like this could be used to reproduce bugs affecting
a particular JVM, servlet container, filesystem layout, or other
details of the (Linux) platform.