*-api-plugin (tests downstream plugins)?

[Olivier Lamy]

Hi
As we use more and more *-api-plugin dependencies, breaking release of
one of those might break a few downstream dependent plugins..
Such this issue https://issues.jenkins.io/browse/JENKINS-70637 which
has broken a few users including myself (that's probably why I'm
writing this email 🤣)
As the *-api-plugin doesn't have any tests there is no way to figure
out potential issues :(
I wonder how we could improve to have some tests with downstream plugins?
maybe some solution "a la" PCT with injecting the upgrade version to
downstream plugins and run the tests?

Thoughts?

cheers
--
Olivier

[Mark Waite]

I like the idea.  Would that have detected the specific problem that prompted the idea?  Are there tests in plugins that depend on the API plugin that would have detected the issue?

If there aren't tests that would have detected the issue, wouldn't it be better to add tests to the API plugin to prevent that problem in the future?  

[Olivier Lamy]

On Mon, 20 Feb 2023 at 00:54, Mark Waite <mark.ea...@gmail.com> wrote:
>
> I like the idea. Would that have detected the specific problem that prompted the idea? Are there tests in plugins that depend on the API plugin that would have detected the issue?

tbh not sure we will detect all possible issues :) but at least we
could prevent some.

>
> If there aren't tests that would have detected the issue, wouldn't it be better to add tests to the API plugin to prevent that problem in the future?

why not but I'm not sure how we can collect all usage in downstream plugins.

I'm not quite sure how to implement this technically because
downstream plugins may use the api-plugin as direct dependency with
version or using the bom so this means we would have to inject an
incremental bom or change version.
how to detect all downstream plugins? manual actions in the plugin-api
by adding manually in Jenkinsfile the downstream plugins? having some
scanning on artifactory to detect links? (maybe something already
exists in Artifactory via some provided API).

>
> On Friday, February 17, 2023 at 9:03:44 PM UTC-7 olive...@gmail.com wrote:
>>
>> Hi
>> As we use more and more *-api-plugin dependencies, breaking release of
>> one of those might break a few downstream dependent plugins..
>> Such this issue https://issues.jenkins.io/browse/JENKINS-70637 which
>> has broken a few users including myself (that's probably why I'm
>> writing this email 🤣)
>> As the *-api-plugin doesn't have any tests there is no way to figure
>> out potential issues :(
>> I wonder how we could improve to have some tests with downstream plugins?
>> maybe some solution "a la" PCT with injecting the upgrade version to
>> downstream plugins and run the tests?
>>
>> Thoughts?
>>
>> cheers
>> --
>> Olivier
>

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/b9f57672-7cae-4c4d-8486-2f4828aa8d2an%40googlegroups.com.

--
Olivier Lamy
http://linkedin.com/in/olamy

[Mark Waite]

On Sunday, February 19, 2023 at 2:50:44 PM UTC-7 Olivier Lamy wrote:

On Mon, 20 Feb 2023 at 00:54, Mark Waite  wrote:
>
> I like the idea. Would that have detected the specific problem that prompted the idea? Are there tests in plugins that depend on the API plugin that would have detected the issue?

tbh not sure we will detect all possible issues :) but at least we
could prevent some.

>
> If there aren't tests that would have detected the issue, wouldn't it be better to add tests to the API plugin to prevent that problem in the future?

why not but I'm not sure how we can collect all usage in downstream plugins.

I'm not quite sure how to implement this technically because
downstream plugins may use the api-plugin as direct dependency with
version or using the bom so this means we would have to inject an
incremental bom or change version.
how to detect all downstream plugins? manual actions in the plugin-api
by adding manually in Jenkinsfile the downstream plugins? having some
scanning on artifactory to detect links? (maybe something already
exists in Artifactory via some provided API).

I think that the necessary information may already be available from the update center, since it is visible on the dependencies page of the plugins site.  Though, possibly I'm not understanding the issue you're raising...

https://plugins.jenkins.io/kubernetes-client-api/#dependencies shows that there are four plugins that depend on the Kubernetes client API plugin:

Kubernetes
Kubernetes Credentials
Kubernetes Credentials Provider
OpenShift Sync

https://plugins.jenkins.io/okhttp-api/#dependencies shows the plugins that depend on the okhttp API plugin:

Azure AD
CloudBees Docker Hub/Registry Notification
CloudBees Feature Management
CodeThreat
GitHub
GitHub API
GitHub Branch Source
Helix ALM Test Management
LoadRunner Cloud
OpenTelemetry
Performance Signature: Dynatrace
Performance Signature: Dynatrace AppMon
Pipeline Input Step Notification
REST List Parameter

 

[Olivier Lamy]

On Mon, 20 Feb 2023 at 08:38, Mark Waite <mark.ea...@gmail.com> wrote:
>
>
>
> On Sunday, February 19, 2023 at 2:50:44 PM UTC-7 Olivier Lamy wrote:
> On Mon, 20 Feb 2023 at 00:54, Mark Waite wrote:
> >
> > I like the idea. Would that have detected the specific problem that prompted the idea? Are there tests in plugins that depend on the API plugin that would have detected the issue?
>
> tbh not sure we will detect all possible issues :) but at least we
> could prevent some.
>
> >
> > If there aren't tests that would have detected the issue, wouldn't it be better to add tests to the API plugin to prevent that problem in the future?
>
> why not but I'm not sure how we can collect all usage in downstream plugins.
>
> I'm not quite sure how to implement this technically because
> downstream plugins may use the api-plugin as direct dependency with
> version or using the bom so this means we would have to inject an
> incremental bom or change version.
> how to detect all downstream plugins? manual actions in the plugin-api
> by adding manually in Jenkinsfile the downstream plugins? having some
> scanning on artifactory to detect links? (maybe something already
> exists in Artifactory via some provided API).
>
>
> I think that the necessary information may already be available from the update center, since it is visible on the dependencies page of the plugins site. Though, possibly I'm not understanding the issue you're raising...

Nice if the information is already available (I was not sure)

My issue is (but I think you have it? :) ):
- one of those *-api-plugin is upgraded with a new version of what it
is wrapped, released then immediately available in the plugin center.
- user just upgrade this because it's available in the update center
- boom downstream plugins are failing because the -api-plugin has
upgraded to a version non compatible with downstream plugins

>
> https://plugins.jenkins.io/kubernetes-client-api/#dependencies shows that there are four plugins that depend on the Kubernetes client API plugin:
>
> Kubernetes
> Kubernetes Credentials
> Kubernetes Credentials Provider
> OpenShift Sync

Is it this information available in a json file or something else?

>
> https://plugins.jenkins.io/okhttp-api/#dependencies shows the plugins that depend on the okhttp API plugin:
>
> Azure AD
> CloudBees Docker Hub/Registry Notification
> CloudBees Feature Management
> CodeThreat
> GitHub
> GitHub API
> GitHub Branch Source
> Helix ALM Test Management
> LoadRunner Cloud
> OpenTelemetry
> Performance Signature: Dynatrace
> Performance Signature: Dynatrace AppMon
> Pipeline Input Step Notification
> REST List Parameter

this mean any changes in okhttp-api should run tests of all of those
plugins by injecting an incremental version of okhttp-api

>
>
> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/39c97990-afb7-4add-8aba-779ac9b98af8n%40googlegroups.com.

[Mark Waite]

On Sunday, February 19, 2023 at 7:32:47 PM UTC-7 Olivier Lamy wrote:

On Mon, 20 Feb 2023 at 08:38, Mark Waite  wrote:
>
>
>
> On Sunday, February 19, 2023 at 2:50:44 PM UTC-7 Olivier Lamy wrote:
> On Mon, 20 Feb 2023 at 00:54, Mark Waite wrote:
> >
> > I like the idea. Would that have detected the specific problem that prompted the idea? Are there tests in plugins that depend on the API plugin that would have detected the issue?
>
> tbh not sure we will detect all possible issues :) but at least we
> could prevent some.
>

I think that we need more evidence that the technique would detect issues.  I'm not sure how the api plugins are much different from other plugins that provide an API.

The bouncycastle API plugin would lead to tests of 800+ plugins, with those plugins in various states of use (some with no installations, many with very few installations).  It is clearly an API plugin.  I am not sure that the work to test drive in 800 plugins is worth the benefit gained from that testing.  I would expect most of the work in that effort would be preparing the consumer plugin to use the new release of the API plugin.

The git client plugin provides the JGit API but is not an API plugin in the same way that pure API plugins are.  It would lead to tests of 29 plugins, with the same issue that many of those plugins have very few installations and may not even compile with a new version of the git client plugin.

I see many challenges to be addressed and am not yet persuaded of the number of cases that the technique will detect.

 

[Basil Crow]

Yes, most library plugins do not have sufficient test coverage. They
should all have at least one smoke test based on RealJenkinsRule,
since JenkinsRule is not a realistic linkage environment. Testing
downstream consumers from a PR build is essentially the second half of
JENKINS-45047 and would be a welcome enhancement.

[Jesse Glick]

On Sun, Feb 19, 2023 at 4:50 PM Olivier Lamy <olive...@gmail.com> wrote:

I'm not quite sure how to implement this technically because
downstream plugins may use the api-plugin as direct dependency with
version or using the bom so this means we would have to inject an
incremental bom or change version.

 

https://issues.jenkins.io/browse/JENKINS-45047 gives the basic idea. The basic machinery is already used from `jenkinsci/bom`. See the second bullet point in https://github.com/jenkinsci/maven-hpi-plugin/pull/66#issuecomment-1110203758 for discussion.

how to detect all downstream plugins?

As Mark mentions, that would rarely be practical. We could test some representative downstream plugins.

A different approach would be to make the UC generator continue to pluck the latest version from Artifactory for “long-tail” plugins, but for plugins managed in `bom`, pick up their versions from the last stable `master` build or whatever. That would ensure that regressions among a large set of commonplace plugins do not accidentally slip onto the UC before they are corrected. This would not have helped in this case, though, since `kubernetes` has a very specialized test suite that is not covered by `bom`. Similarly for `artifact-manager-s3` vs. AWS SDK, etc.