Re: Jenkins acceptance test Harness release history

[oliver gondža]

Hi, That definitely sounds interesting! See my replies inline:

On Thu, 29 Oct 2015 14:26:59 +0100, Aditya Nisal
<adity...@googlemail.com> wrote:

> Hello Ulli!
>
> Thank you for your valuable feedback! Next time onwards I would
> definitely
> ask such questions in the dev mailing list. I did post my questions in
> the
> #jenkins IRC channel but decided to mail directly the repository
> contributors, since I did not get any response there.
>
> My thesis aims to analyze the robustness and stability of selenium tests
> over the version history of Jenkins. Thus, I am going to take 3 major
> versions of Jenkins (1.4xx,1.5xx,1.6xx) and their respective minor
> releases, and execute the acceptance test suite against them. However I
> was
> wondering if after a major version is released, if the test-cases need a
> significant modification in order to maintain the same coverage/passed
> tests. Hence I would like to know if there are any such points (perhaps
> identified by a particular tag) when the tests underwent a major
> overhaul?

Some remarks:

- Make sure to pick LTS releases as those tend to be lot more stable than
weekly ones. Comparing weekly with LTS can give seriously biased results.
- Note that there are some test annotated with @Since("X") that does not
run with Jenkins version older than X as covered use-case was not
supported yet. Later versions can then produce more successful tests
without improving the stability.
- It can be even more tricky with bundled plugin version constrains
@WithPlugins("Y@X")
- Make sure to run the suite with docker daemon on not to skip docker
based tests.
- There are no major version of Jenkins (in traditional sense), though the
core/plugin evolution breaks ATH occasionally.
- Usually new tests are developed with "recent" versions in mind and
whoever needs to run ATH against older version contributes his/her time to
keep ATH running on that version.
- Last version I used ATH against was 1.480, I am afraid it would take a
significant effort for you to make it running for anything older.
- Note that such "incompatibilities" between ATH and Jenkins will be
more common for older releases. Again, newer versions can appear more
stable than old ones because of that.
- Tests in ATH are rather "sparse" meaning it support only a fracture of
Jenkins use-cases (compared to jenkins-test-harness for instance). I am
not sure how reliable the results will be.

> I am also planning to publish the results after my experiment and inform
> about them in the irc channel, in case someone would like to use a
> feedback.

jenkinsci-dev would be definitely interested in that!

> On 29 October 2015 at 13:34, Ullrich Hafner <ullrich...@gmail.com>
> wrote:
>
>> Hi Adi,
>>
>> first of all: such questions should be asked in the dev mailing list so
>> others can participate (or more importantly see the answers later on).
>>
>> Am 29.10.2015 um 11:20 schrieb Aditya Nisal
>> <adity...@googlemail.com>:
>>
>> Hello all!
>>
>> My name is Adi and I am a MSc Computer Science student from Saarland
>> University, Germany. As a part of my master's thesis I am analyzing
>> Jenkins
>> Selenium tests (https://github.com/jenkinsci/acceptance-test-harness/).
>> I had a question regarding the release history of this test suite. As
>> per
>> the releases on the github repository, there are 16 releases (tags),
>> however, I did not understand if there is any mapping /relationship
>> between
>> Jenkins' releases and the acceptance tests' releases. I investigated the
>> pom.xml of all releases, but the
>> parent pom always points to Jenkins version 1.32.
>>
>> <parent>
>> <groupId>org.jenkins-ci</groupId>
>> <artifactId>jenkins</artifactId>
>> <version>1.32</version>
>> </parent>
>>
>>
>>
>> This dependency is only there to import some basic properties which are
>> used both by Jenkins and the test suite. Actually there is (almost) no
>> relationship from the test suite to a Jenkins release. I.e. the tests
>> should run for several releases of Jenkins. And in fact we already run
>> the
>> same suite for Jenkins LTS and DEV. (Internally we specify for test
>> cases
>> for which release a test case is created for.)
>>
>> Moreover, I could not map the tests with Jenkins by comparing the
>> release-timestamps, since some releases are 3-4 months apart, while some
>> are in the span of few weeks.
>>
>>
>> Releases of the test suite are used only internally by CloudBees. The
>> Jenkins project always uses the last commit from the master branch when
>> running the test suite. So just ignore these releases. (Jenkins releases
>> are on a weekly basis.)
>>
>> Best regards from Germany, too :-)
>>
>> Viele Grüße,
>>
>> Ulli
>>
>> I would be more than thankful if you could give me some insights about
>> the
>> relationship between the two.
>>
>> Best regards from Germany,
>> Adi
>>
>>
>>


--
oliver