Opening Plugin Compatibility Tester

[Oliver Gondža]

Hi,

For a period of time I see several people from CB contributing to PCT
itself and addressing problem in plugins pointing out PCT related
problems. However, last time I checked, I have not found a compelling
documentation on how to run it or any public Jenkins job where people
can check the results. Hence I presume CB is running those tests internally.

While I do appreciate people are working on improving the testing and
the code itself, I would like all that could be opened for people to
reuse and run it on ci.jenkins.io (ideally) where everybody can check
for the results. (Consider your are the maintainer that received a PR
that purports to fix something you have no evidence of being broken
referring to a TLA-named testsuite you have never heared of and have no
easy way to checking for yourself...)

Is there any chance you can update the wiki and share your Jenkinsfiles?

Thanks
--
oliver

[Robert Sandell]

that is strange, I remember copying something from a freestyle job on one of the Jenkins project's masters for the initial draft of our internal PCT job. But I think it was disabled at the time probably due to it being flaky and using up too much resources for the project.

Perhaps it got lost in the migration to ci.jenkins.io?

/B

--
oliver

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/95cc19a3-b5ce-ae51-8366-c3faf840887d%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Robert Sandell

Software Engineer

CloudBees Inc.

[Jesse Glick]

On Tue, May 9, 2017 at 4:11 AM, Oliver Gondža <ogo...@gmail.com> wrote:
> a PR that purports to fix
> something you have no evidence of being broken referring to a TLA-named
> testsuite you have never heared of and have no easy way to checking for
> yourself

FWIW I have asked people filing such PRs to include some background
information as well as instructions to reproduce the failure locally.
In most (not all) cases this comes down to something like

mvn clean test-compile && mvn -Djenkins.version=2.x.y surefire:test

or even

mvn -Djenkins.version=2.x.y clean test

The last can be kept passing by writing your `Jenkinsfile` as

buildPlugin(jenkinsVersions: [null, '2.x.y'])

for whatever LTS line. There are also some cases where failures are
due to plugin-to-plugin dependency bumps, which `buildPlugin` does not
currently support as a testing option.

As to whether there can be a public Jenkins job running precisely the
same set of tests as CloudBees runs internally, I am doubtful, since
the particular set of core & plugin versions being tested is defined
by the versions promoted in CloudBees products, and having this as a
scheduled job would make little sense since a test run is tied to a
change to that revision matrix.

Now what *would* be extremely useful, if someone cares to take the
time to set it up and maintain it, is a job on ci.jenkins.io which
periodically (daily?) runs the PCT against some defined but
ever-increasing (transitively closed) set of plugins, using trunk
snapshot versions of core and all included plugins. If failures are
evaluated and assigned promptly, this would alert developers to
compatibility regressions they need to either fix or acknowledge,
before releases are even made (albeit after faulty PRs are merged—a
harder problem to solve I think).

[eguti...@cloudbees.com]

Hi Oliver,

As Jesse comments:

• We are trying to be as clear as possible on that kind of PRs, so that both the problem and the fix can be understood and acknowledged by the maintainer and whoever might evaluate such PR.
  
• It's not easy to directly translate what we are doing internally as PCT is designed to use the plugins offered by UC, what would probably lead to ever lasting executions. Also we tight PCT runs to a certain set of core and plugin versions.

Being said that, I agree that someone could miss some context or a more in-depth explanation both from a technical (how to execute it) and a conceptual perspective (what PCT intends to be useful for). So we will try to:

• Put in place such information so that at least people can understand better the whole PCT itself.
• Analyze what needs to be done in PCT to allow executing it with the proper conditions (for example a set of plugins) and put in place some work to make it feasible.

Thanks!

[Oleg Nenashev]

Hi,

Just to bump the threads.
As a part of JEP-200 I needed to run PCT against a number of plugins recommended in the installation wizard.

I spent some time studying PCT and finally created a pull-request with the documentation update and Docker image. It should simplify the custom PCT runs. If anybody is interested, the pull request is here: https://github.com/jenkinsci/plugin-compat-tester/pull/53

BR, Oleg

среда, 10 мая 2017 г., 9:08:25 UTC+2 пользователь eguti...@cloudbees.com написал: