LTS RC test plan update

[Oliver Gondža]

As I look at the
https://wiki.jenkins-ci.org/display/JENKINS/LTS+2.7.x+RC+Testing, it
gets less and less relevant with time. I only ever deleted scenarios
that was covered by automated tests but there are plugins/use cases
where noone cared to report acks/problems for over a year. All in all, I
see no reason to have them in the test plan.

On the other hand there might be plenty of things we would like to have
in (the automated or manual test plan).

- Suggested plugins installed by wizard.
- Use cases from https://jenkins.io/solutions/.
- ???

I wonder if there are any other suggestion or objection to this plan. I
intend to come up with some more updated list and let users know we are
testing something they can actually be interested in getting more
testers hopefully.

--
oliver

[Mark Waite]

That sounds good to me.  Simplifying that page will be a good thing.

I've constructed a series of Docker images which contain bug verification jobs that are interesting to the git plugin.  Currently it includes 72 jobs that verify specific bugs have not been reintroduced, with an additional 20+ jobs verifying various forms of authentication to interesting git service providers like GitHub, BitBucket, Gitlab, and Assembla.

Are you interested in links to the verification Docker definitions (for those images that I'm willing to share publicly)?

I've found it quite helpful to use a Jenkins job to confirm that a specific git plugin bug is resolved.  It then allows me to run that same condition on multiple platforms, with parallel execution (depending on the number of slaves I have running), and has been helpful discovering unexpected surprises in my configurations and in those tests.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/83d74819-11b1-692b-45e2-e2f23b960d8f%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Oliver Gondža]

On 2016-06-30 16:26, Mark Waite wrote:
> That sounds good to me. Simplifying that page will be a good thing.
>
> I've constructed a series of Docker images which contain bug
> verification jobs that are interesting to the git plugin. Currently it
> includes 72 jobs that verify specific bugs have not been reintroduced,
> with an additional 20+ jobs verifying various forms of authentication to
> interesting git service providers like GitHub, BitBucket, Gitlab, and
> Assembla.
>
> Are you interested in links to the verification Docker definitions (for
> those images that I'm willing to share publicly)?
>
> I've found it quite helpful to use a Jenkins job to confirm that a
> specific git plugin bug is resolved. It then allows me to run that same
> condition on multiple platforms, with parallel execution (depending on
> the number of slaves I have running), and has been helpful discovering
> unexpected surprises in my configurations and in those tests.
>
> Mark Waite

It is good to hear you still look at the LTS RCs, for sure. I would love
if your work ware living and running somewhere publicly so people can
contribute and see the results. Though it seems you approach is
fundamentally different than the one used by ATH and project infra is
not ready to run docker containers (unless that has changes recently).

As a minimum, please consolidate the path of the wiki page that refers
to git plugin to something that makes sense w.r.t. what you tests. Also,
I really miss your green +1 in there so feel free to have just one item
on the page but please, let us know the tests are passing at your end.

Thanks
--
oliver

[Mark Waite]

Yes, my technique is quite different than the acceptance test harness.  I've had poor experiences with test automation that tries to drive a web browser and detect problems.  In my case, I like the interactive nature of the verification jobs and I really like that Jenkins distributes them to as many (or as few) agents as are available, without any real thought from me.

Running the docker instance and placing the assertions inside Jenkins jobs (with fail or unstable an indication of a possible bug) has worked well thus far.  It is not as fine-grained as the acceptance test harness, but it meets my needs while I am evaluating pull requests to the git plugin and the git client plugin, and helps me learn more about various Jenkins components working together.  Pipeline has been especially fun as I've now learned a little bit of groovy scripting.

The public version of my docker definitions are available at https://github.com/MarkEWaite/docker/tree/lts-with-plugins (Jenkins 1.651.3) and at https://github.com/MarkEWaite/docker/tree/lts-oldest-with-plugins (Jenkins 1.609.3).  My version which uses the release candidate is on a private github repository so that I can include various credentials and other sensitive information in the Docker instance.

Mark Waite