[GSOC] Jenkins REST API Client Pipeline Wrapper Plugin Proposal

[martinda]

Hi,

We currently have a Google Summer of Code proposal aimed at wrapping a Jenkins REST API Client into a Jenkins Pipeline Plugin. There are a few jenkins rest api clients which can be used to make this possible:

1. https://github.com/cdancy/jenkins-rest
   
2. https://github.com/jenkinsci/java-client-api
   
3. https://github.com/cliffano/swaggy-jenkins
   

I am reaching out to the community to probe interest, feedback on the proposal, comments and to recruit mentors for this project. Being a mentor is open to everyone.

Thanks,

Martin d'Anjou

Jenkins GSoC 2019 Org Admin

[Jesse Glick]

I am afraid my immediate reaction to both this and the Artifactory
REST plugin proposal is that we would be better off without such
plugins. As an “emeritus maintainer” of `workflow-cps` I can say that
this kind of feature is rather risky and prone to weird bugs, and
would burden efforts to produce an alternate execution engine. Writing
an external script (in Python or whatever) is much more reliable and
testable:

withCredentials([string(credentialsId: 'other-jenkins-api-token',
variable: 'TOKEN')]) {
sh 'python jenkins-tricks.py'
}

Pipeline steps should (a) have simple, databinding-compatible input
types and ideally only primitive/`String` output types; (b) be limited
to features which involve interacting directly with the host Jenkins
service in a way that would be impractical to simulate otherwise.

[martinda]

I have marked the google docs for all three as being discussed here. We may have to pull out those proposals if they prove to be problematic.

burden efforts to produce an alternate execution engine

The two execution engines I know of are the scripted pipeline and the descriptive pipeline. Is there talks about a newer one?

(b) be limited to features which involve interacting directly with the host Jenkins 
service in a way that would be impractical to simulate otherwise. 

The http request plugin is "untestable" by that definition, yet it is useful to talk to external services.

Sometimes, it can be useful to talk to a remote Jenkins master from another Jenkins master.

But if you say that there are problems with returning non-simple data types, are you also saying that there are problems providing methods that return simple data types?

e.g. jenkinsRest.api().systemApi().version() -> return the version number as a String?

Thanks,

Martin

[Jesse Glick]

On Fri, Dec 28, 2018 at 5:49 PM martinda <martin....@gmail.com> wrote:
>> burden efforts to produce an alternate execution engine
>
> The two execution engines I know of are the scripted pipeline and the descriptive pipeline. Is there talks about a newer one?

No, that is the same execution engine—`workflow-cps` directly
implements Scripted, and Declarative is a kind of preprocessor for
that. An alternate engine would replace the on-master Groovy
interpreter.

> if you say that there are problems with returning non-simple data types, are you also saying that there are problems providing methods that return simple data types?
> e.g. jenkinsRest.api().systemApi().version() -> return the version number as a String?

The problem here is (1) using the `GlobalVariable` extension point
(please do not), (2) returning intermediate objects with behaviors.
Such idioms rely on particularities of `workflow-cps` (and do not even
work well in Declarative).

[martinda]

On Monday, December 31, 2018 at 9:34:58 AM UTC-5, Jesse Glick wrote:

On Fri, Dec 28, 2018 at 5:49 PM martinda <martin....@gmail.com> wrote:
>> burden efforts to produce an alternate execution engine
>
> The two execution engines I know of are the scripted pipeline and the descriptive pipeline. Is there talks about a newer one?

No, that is the same execution engine—`workflow-cps` directly
implements Scripted, and Declarative is a kind of preprocessor for
that. An alternate engine would replace the on-master Groovy
interpreter.

Sounds like a large complex project. Maybe someone from the Pipeline Authoring SIG could comment?

 

> if you say that there are problems with returning non-simple data types, are you also saying that there are problems providing methods that return simple data types?
> e.g. jenkinsRest.api().systemApi().version() -> return the version number as a String?

The problem here is (1) using the `GlobalVariable` extension point
(please do not), (2) returning intermediate objects with behaviors.
Such idioms rely on particularities of `workflow-cps` (and do not even
work well in Declarative).

Ok, so is there a way forward for plugins of that type (those that intent to wrap REST APIs of external services)?

Thanks,

Martin 

[Jesse Glick]

On Sat, Jan 5, 2019 at 12:23 AM martinda <martin....@gmail.com> wrote:
>>> e.g. jenkinsRest.api().systemApi().version() -> return the version number as a String?
>

> so is there a way forward for plugins of that type (those that intent to wrap REST APIs of external services)?

Besides not existing? At a minimum, use no `GlobalVariable`s or any
other `workflow-cps` dep, and use the `Step` API as it was designed:
so that the arguments are straightforward uses of Jenkins databinding
(comprehensible by `structs`). If you want to be compatible with
Declarative Pipeline, there must be no return value, which rules out
most such plugins unless they merely set up authentication environment
variables and/or `$PATH`. If you are limiting yourself to Scripted,
then any return values should still be simple data like primitives,
`String`s, `List`s, or `Map`s.

As an aside, for the special case that the Jenkins instance being
queried is the same one the build is running on, I prototyped

https://github.com/jenkinsci/workflow-basic-steps-plugin/pull/71

which does not make any actual HTTP call (so needs no stored
credentials), and is designed to be usable on other plausible
execution engines running something like Scripted syntax.

[Martin d'Anjou]

Besides not existing?

So you are really saying that Jenkins needs a new type of execution engine in order to support plugins that return non-simple data.

This is way beyond my level of expertise. Has anything been attempted or discussed in that area?

 

If you are limiting yourself to Scripted,
then any return values should still be simple data like primitives,
`String`s, `List`s, or `Map`s.

Is the requirement asking for Serializable?

I am confused by Lists and Maps which are collections (sort of nested).

I looked at the Step Javadoc and Step API Documentation but could not find that information.

Is the list of data types that are supported documented?

Lastly, I remember when I converted the http request plugin to pipeline, I created a response object returning only simple data types, I guess that is what you mean we should do.

BR,

Martin

[Jesse Glick]

On Tue, Jan 8, 2019 at 9:17 AM Martin d'Anjou <martin....@gmail.com> wrote:
> So you are really saying that Jenkins needs a new type of execution engine in order to support plugins that return non-simple data.

No no, I am saying that Jenkins needs a new type of execution engine,
and the proposed replacement would have a hard time supporting steps
that return non-simple data.

> Is the requirement asking for Serializable?
> I am confused by Lists and Maps which are collections (sort of nested).

There is not a fixed list of types for now, but as a rule, do not
return any objects with behavior.

[martinda]

I am re-working the 3 REST API GSoC proposals...

How does this example look like in terms of respecting the current execution engine requirements:

bitbucketClient.withServer(url: “foo://bitbucket”, project: project, repo: repo, crendentialsId: “mybitbucket”) {
    def response = bitbucketClient.pullRequestApi(‘changes’, prNumber)
    echo response.errors // a list
    echo response.errors[0].context // a string
    echo response.errors[0].message // a string
    echo response.changes // a list
    echo response.changes[0].path // a string
    echo response.changes[0].srcPath // a string
}

The response object only ever holds simple data types.

Other question: Is it okay for the step to have multiple methods, but they only return simple data types?

For example: bitbucketClient.pullRequestApi(prNumber, 'changes'); bitbucketClient.branchApi('list')

Martin

[Marky Jackson]

This looks good and yes, the step can have multiple methods.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/285c7bae-5e53-4f69-aa6b-28f00186315a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jesse Glick]

On Wed, Jan 16, 2019 at 9:43 PM martinda <martin....@gmail.com> wrote:
> bitbucketClient.withServer(url: “foo://bitbucket”, project: project, repo: repo, crendentialsId: “mybitbucket”) {

Whatever you are writing here, it is not a `Step`. A step is a
function (implemented for purposes of Groovy as a method with the
implicit receiver being the `Script`) taking named arguments according
to Jenkins databinding restrictions.

[martinda]

At a minimum, use no `GlobalVariable`s or any
other `workflow-cps` dep, and use the `Step` API as it was designed

What about JFrog's artifactory plugin, it uses a workflow-cps dependency...?

I don't understand... "GlobalVariable" exists but plugin developers should not use it?

Martin

[Jesse Glick]

On Sat, Jan 26, 2019 at 1:42 PM martinda <martin....@gmail.com> wrote:
> What about JFrog's artifactory plugin, it uses a workflow-cps dependency...?

And has had numerous problems as a result.

> "GlobalVariable" exists but plugin developers should not use it?

No, generally not.

https://javadoc.jenkins.io/plugin/workflow-cps/org/jenkinsci/plugins/workflow/cps/GlobalVariable.html

has some discussion.

[sbara...@gmail.com]

Hi Martin,

Are we going to rewrite codes by using any one of the 3 plugins or write from scratch?? If its going to be from scratch, I find that its always best to cover all aspects of Jenkins to be covered under REST API. Similar to Bitbucket APIs, like connecting the configuration till enabling and building pipeline or proceeding via the APIs. As it will make the Jenkins more powerful and easier to integrate it with third-parties.

[martinda]

Hi,

The student is free to propose different solutions and different implementations in the proposal they submit. Another approach for this type of coding project is to use OpenAPI (but it probably did not exist at the time). Regarding the reason these ideas are in GSoC, is that I was impressed by those libraries when I started working with them. They are concise (probably because they are based on Apache JClouds), clear, well written, and come with live tests and unit tests, so I thought it would be good to wrap them as plugins for Jenkins, hence I suggested these ideas for GSoC this year.

Martin

[Barathwaja S]

Hi,

I agree that OpenAPI has a standard on writing like how the API end points will look like the same way is that bitbucket endpoints too.  And I'm having an issue while writing a custom API endpoint I get below error like

Stapler processed this HTTP request as follows, but couldn't find the resource to consume the request

-> evaluate(<hudson.model.Hudson@4273c8a3> :hudson.model.Hudson,"/json/test")
-> evaluate(((StaplerProxy)<hudson.model.Hudson@4273c8a3>).getTarget(),"/json/test")
-> evaluate(<hudson.model.Hudson@4273c8a3>.getDynamic("json",...),"/test")
            hudson.mo...@4273c8a3.getDynamic("json",...)==null. Back tracking.
-> evaluate(((StaplerFallback)<hudson.model.Hudson@4273c8a3>).getStaplerFallback(),"/json/test")
-> evaluate(<hudson.model.AllView@119d39b0[view/All/]> :hudson.model.AllView,"/json/test")
-> evaluate(<hudson.model.AllView@119d39b0[view/All/]>.getDynamic("json",...),"/test")
            hudson.model.AllView@119d39b0[view/All/].getDynamic("json",...)==null. Back tracking.

How to add wrapper on top to access ?? Any Idea??

Thanks and Regards,

Barathwaja S

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/2204c220-3d86-4130-88fc-b7cd54d51102%40googlegroups.com.