Pipeline View Plugin

[Sergei Egorov]

Hey everyone, 

Yesterday I decided to publish sources for my pipeline visualization plugin: https://github.com/bsideup/jenkins-pipeline-view

What's cool about it? 

First of all, it can handle any graph provided by pipeline because it uses very powerful JS graph library to position steps. 

Also, it's a React application, written in ES7 (not even ES6!), with ImmutableJS and RxJS inside - so hipsters will be happy :D But in fact, it means that it's damn easy to develop this plugin and provide more functionality. 

I use Webpack to bundle everything (JS, CSS, fonts, images, SVG icons) in one single pipe.js file. No Jenkins JS Modules, no conflicts, no impact on others. Even CSS will not conflict because of CSS-modules ( http://glenmaddern.com/articles/css-modules )

All icons are SVG ones and look good on any screen, retina or not, and any zoom level.

I use Jackson on the backend side to serialize FlowNodes and their actions. Why Jackson? Because it was much easier to implement serialization of selected (non-exposed) fields and class info included compared to Stapler. I saw something were done about classinfo in Stapler, but at the moment of creation of this plugin, it wasn't delivered to Jenkins core yet. Also, almost none of the Pipeline actions are @Expose-d.

I use gradle-jpi-plugin instead of Maven because it's much easier to describe some complex build process with Gradle, especially when frontend build is involved. In fact, it's just one line:

https://github.com/bsideup/jenkins-pipeline-view/blob/954b895b6574cdf34815ff94a4a8db3ad3811aeb/build.gradle#L61

Future development

Will I continue to develop it? Definitely! Here, at ZeroTurnaround, we use Jenkins a lot, and eventually we will migrate to the pipelines, and proper visualization of the process is a must for us. And feel free to contribute as well, it's really a good chance to learn modern JS stack as well :)

Best regards,

Sergei Egorov

[Kanstantsin Shautsou]

Cool work!

[Antonio Muñiz]

Nice! I'm trying to build it but after `./gradlew build` the build/dist directory is empty.

The build finishes wit success, but I see this error in the build log: 

ERROR in ./app/index.jsx

Module build failed: Error: Cannot find module 'es2015'

Something related to i18n?

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/61fe5fb1-6b5f-41b8-bdb0-f5d04c5b9f54%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Antonio Muñiz

Software Engineer
CloudBees, Inc.

[Sergei Egorov]

Het Antonio,

Thanks for the report! I will clean my local installation and re-run build to make sure that It will work for you as well

I will report in a few minutes

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAJc7kzS28M8q6EruxGXE%2B8yC9vELsCuW-qXFLS1JZG-8ASge%3Dg%40mail.gmail.com.

[Sergei Egorov]

Ah, yes, definitely my bad. Forgot to include peer dependency. Should be fine now if you pull the latest changes. Sorry :)

[Sergei Egorov]

Also, just noticed a typo in README: build/libs/pipeline-view.hpi instead of "build/dist". 

[Jesse Glick]

Looks nice! A nit about the README: it should emphasize that while in
_dev mode_ you need to manually run a separate service and browse to
another URL, the fully built plugin `*.hpi` is self-contained.

On Mon, Mar 21, 2016 at 2:52 AM, Sergei Egorov <bsi...@gmail.com> wrote:
> Why Jackson? Because it was much easier to implement serialization of selected (non-exposed) fields and class info included compared to Stapler. I saw something were done about classinfo in Stapler, but at the moment of creation of this plugin, it wasn't delivered to Jenkins core yet. Also, almost none of the Pipeline actions are @Expose-d.

BTW there is no apparent purpose in `PipelineViewApi` extending `Api`,
since you are just defining a custom web method anyway, which could
have put directly in `PipelineViewAction`. Maybe I am missing
something here.

[Sergei Egorov]

Hi Jesse,

Ok, will note it in README. Since all Jenkins plugins are self-contained I was thinking that it doesn't worst a mention that outside of the dev mode it differ :)

About the API - well, you're right, I guess... I just prefer to split responsibility into the separate classes, this is why I didn't want to put API method in Action class and decided to provide API instead. Not sure if it's best practice in Jenkins or not thou :)

Anyway, thanks for your notes, I will take them into the account :) 

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0Te6a5_pzm0HnKiz3eGPc7cv%2BWZTE8ibJCB%3DdeOk4_2Q%40mail.gmail.com.

[Michael Neale]

Very cool, and I like the use of react!

[Michael Neale]

As luck would have it, I notice this is tripped by by having global lib variables added in (basically, DSLs). Is there a JIRA project for this? 

(not a huge thing, I can't imagine many would run into this, but it looks like it gets upset if not all the things are serializable). 

Caused by: com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: org.jenkinsci.plugins.workflow.cps.nodes.StepEndNode["typeDisplayName"])

at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:141)

at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)

at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2909)

at com.github.bsideup.jenkins.pipeline.ui.view.PipelineViewApi.doNodes(PipelineViewApi.java:75)

... 89 more

Caused by: java.lang.IllegalArgumentException: (was java.lang.NullPointerException) (through reference chain: org.jenkinsci.plugins.workflow.cps.nodes.StepEndNode["typeDisplayName"])

at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:3459)

at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:3378)

at com.github.bsideup.jenkins.pipeline.ui.view.PipelineViewApi$2.apply(PipelineViewApi.java:78)

at com.github.bsideup.jenkins.pipeline.ui.view.PipelineViewApi$2.apply(PipelineViewApi.java:75)

at com.google.common.collect.Iterators$8.next(Iterators.java:812)

at com.fasterxml.jackson.databind.ser.std.IterableSerializer.serializeContents(IterableSerializer.java:89)

at com.fasterxml.jackson.databind.ser.std.IterableSerializer.serialize(IterableSerializer.java:74)

at com.fasterxml.jackson.databind.ser.std.IterableSerializer.serialize(IterableSerializer.java:12)

at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)

... 92 more

[Sergei Egorov]

Hi Michael,

Wow, interesting finding! 

I checked with the sources and looks like it's not an issue of the plugin itself, but StepEndNode. Jackson in plugin configured to serialize only pre-configured fields/methods, and any new object in the domain will not affect it.

What I see here is that StepEndNode::getTypeDisplayName throws an NPE. From StepEndNode's source code, I see that there are no checks for getStartNode() result, and for some reason it's null. This is a good place to start if you want to figure out what's wrong in your setup (even better if you can put conditional breakpoint on getTypeDisplayName and "getStartNode() == null", because I see (in code) that it might be null, but getTypeDisplayName is not null-safe.

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/0697c884-91ad-4716-809e-3c40cf5e35da%40googlegroups.com.

[Sergei Egorov]

Actually... getStartNode marked with @Nonnull, as well as getExecution(), so one of them is null. Execution maybe?

[Michael Neale]

Yeah I think it is something else. When I go to look at the "normal" flownode graph I get a different exception (but a NPE). I think this is something amiss with globals and steps with flownodes. I wouldn't worry about it, this is a bit of a contrived edge case for now (in a differnet version of jenkins core I don't seem to see it... so it will take some chasing down, but not worth it for now).

[Michael Neale]

In another case it looks like Execution, yes... never mind (should be easy to fix once identified). I think the jackson stuff just happened to pick that edge case up (I thought I saw a serialization problem and my brain suddenly turned off looking for null pointers! hahaha. 

Nice work!

[Sergei Egorov]

FYI I configured CircleCI for this plugin, so now it's possible to download the latest *.hpi automated build from there:

i.e. Build #7:

https://circleci.com/gh/bsideup/jenkins-pipeline-view/7#artifacts

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/70fea7d1-fb89-4500-802b-9e7ec2cb3fb1%40googlegroups.com.

[Oleg Nenashev]

Why don't you just move it to jenkinsci org?
The plugin is definitely useful, so I think it is reasonable.

P.S: I'm still planning to run the plugin against big pipelines. I'm just in the slowpoke mode :(

BR, Oleg

вторник, 22 марта 2016 г., 14:07:39 UTC+1 пользователь Sergei Egorov написал:

[Sergei Egorov]

For me, it's too early but if you think that it's fine to include it under jenkinsci in the current state then I don't mind. Yet I want to control development of it for now and keep pull request model for it, I hope you understand :)

Sergei

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/08bfae6f-5de0-41e1-8a1f-4daca75fe753%40googlegroups.com.

[Kanstantsin Shautsou]

Let’s try host you under jenkinsci and you will decide later whether it works for you. 

I will handle :)

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/YZAHSXYCfDY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAAJdKXm%2BjmTcQEMim9s2T7Y0JMJc-tZcP64edZDvVDYTjAG8AQ%40mail.gmail.com.

[Daniel Beck]

On 22.03.2016, at 14:42, Sergei Egorov <bsi...@gmail.com> wrote:

> For me, it's too early but if you think that it's fine to include it under jenkinsci in the current state then I don't mind. Yet I want to control development of it for now and keep pull request model for it, I hope you understand :)

We try hard to not interfere with plugin maintainers who want to keep strict control over their plugin development (a CONTRIBUTING.md helps as well). If you respond within two weeks or so to PRs there never should be a situation in which we'd even _ask_ you whether someone else should become co-maintainer. However, if you ever lose interest in the plugin or circumstances prevent you from working on it further, we as the Jenkins project have a way forward if someone volunteers to take over.

Note that you can release the plugin to the experimental update center by releasing a version something-alpha-something or something-beta-something, e.g. 1.0-beta-1, making the plugin available to a less Java-proficient audience while . Some explanation on this is here:

http://jenkins-ci.org/content/experimental-plugins-update-center

[Jesse Glick]

On Tue, Mar 22, 2016 at 2:56 AM, Michael Neale <mne...@cloudbees.com> wrote:
> When I go to look at the "normal" flownode graph I get a different exception (but a NPE).

File a bug with a stack trace and any steps to reproduce you may have.

[Michael Neale]

yes you can have control over it if in jenkinsci github org as long as you make it clear in CONTRIBUTING.md as Daniel says. For it to build in there however, you need to have your build process integrated with maven (which may be the real bottleneck for you). 

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/YZAHSXYCfDY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/08bfae6f-5de0-41e1-8a1f-4daca75fe753%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

-- 

Regards, 

Michael Neale

(twitter: @michaelneale, skype: michael_d_neale)

Cell: +61 423175597

Cofounder & Engineer @ large in CTO office. 

[Jesse Glick]

On Tue, Mar 22, 2016 at 6:47 PM, Michael Neale <mne...@cloudbees.com> wrote:
> For it to build in there however, you need to have your build process integrated with maven

jenkins.ci.cloudbees.com supports Gradle-based plugin builds.

[Michael Neale]

yep, Tyler pointed out to me! Good to know, in that case, all the more reason to have it moved on over and available to a wider audience.

If it is alpha then it will be in the experimental update centre, but if you want wider feedback from more users, you can just release it to the main update centre (just make things clear in the description as to the status), I think in this case it would be just fine. 

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/YZAHSXYCfDY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1oauvfPoeYcjFV6%2BXHoeE8fctH4xhsKq9n4KCLhbM8qA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Sergei Egorov]

Ok, now it's under jenkinsci org: https://github.com/jenkinsci/pipeline-view-plugin

I will prepare it for the release this week :)

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAEuC6Li88QKTV549b9%3DO-HbDqbgMmyQq%2BqdBiA7OAAicCZQw-A%40mail.gmail.com.

[Michael Neale]

That's great - I look forward to it! thanks for moving it over!

[Tom Fennelly]

Nice work Sergei.

[Samuel Van Oort]

Sergei, I love the visualization this provides of the pipeline DAG!

One thing worth mentioning:  I have been working on a scalable/high-performance & parallel-aware FlowAnalyzer for pipeline stage view that is generalized enough to assist with the data you are visualizing.   Since the rest-api portion of stage view is available as a separate plugin from UI, I might suggest consuming that API or its extensible Analyzer algorithm to power this view.  

Advantages: don't have to roll your own API, get a variety of metrics on each FlowNode (pause/execution timing, status, executor used) and stage.  The algorithm also scales to complex flows with thousands of nodes and implements caching for executed runs.  Finally it supports limiting the number of FlowNodes returned for a stage, so one can render a subset where a stage contains a loop (or deeply nested blocks) returning thousands of nodes. 

Thoughts?

[Sergei Egorov]

Hey Samuel,

Yes, initially I started to use Stage View's REST API as a source. But... It's a HAL and doesn't use or provide a way to use embedded HAL resources. It means that I will have to make a lot of queries to load all the data I want. Also, there were some missing parts of the data as well. 

Also, in fact, I consume standard API provided by FlowNodes, I'm just exposing some fields which are not exposed with Stapler, and if that annotations will be added + new Stapler (with class support) supported then Pipeline View will consume standard REST API of the Jenkins without any additional API at all, at least at this moment. 

Of course, we can collaborate and improve API of Stage View to make it consumable by Pipeline View as well. 

BR,

Sergei

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/9b0c2cbd-e6ec-4ae0-9b51-956d92f80b1f%40googlegroups.com.

[Michael Neale]

this would be quite interesting, I think this all belongs in the stage view, somehow. Having powerful APIs is useful for future visualisations as well. 

Currently the stage view is more about stages (hence the name) so some ability to reduce the "noise" of steps is required (I can imagine some folk in a stage view may indeed want to see steps executing, not just a whole stage). 

On Sat, Mar 26, 2016 at 2:29 AM Sergei Egorov <bsi...@gmail.com> wrote:

Hey Samuel,

Yes, initially I started to use Stage View's REST API as a source. But... It's a HAL and doesn't use or provide a way to use embedded HAL resources. It means that I will have to make a lot of queries to load all the data I want. Also, there were some missing parts of the data as well. 

Also, in fact, I consume standard API provided by FlowNodes, I'm just exposing some fields which are not exposed with Stapler, and if that annotations will be added + new Stapler (with class support) supported then Pipeline View will consume standard REST API of the Jenkins without any additional API at all, at least at this moment. 

Of course, we can collaborate and improve API of Stage View to make it consumable by Pipeline View as well. 

BR,

Sergei

On Fri, Mar 25, 2016 at 5:19 PM Samuel Van Oort <svan...@cloudbees.com> wrote:

Sergei, I love the visualization this provides of the pipeline DAG!

One thing worth mentioning:  I have been working on a scalable/high-performance & parallel-aware FlowAnalyzer for pipeline stage view that is generalized enough to assist with the data you are visualizing.   Since the rest-api portion of stage view is available as a separate plugin from UI, I might suggest consuming that API or its extensible Analyzer algorithm to power this view.  

Advantages: don't have to roll your own API, get a variety of metrics on each FlowNode (pause/execution timing, status, executor used) and stage.  The algorithm also scales to complex flows with thousands of nodes and implements caching for executed runs.  Finally it supports limiting the number of FlowNodes returned for a stage, so one can render a subset where a stage contains a loop (or deeply nested blocks) returning thousands of nodes. 

Thoughts?

[Michael Neale]

Sergei - this hasn't been published to any repo that I can see (and the wiki doesn't show it up https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+View+Plugin) - is this deliberate or a mistake? (I don't see any commits to do with release preparing). 

[Kanstantsin Shautsou]

I think README can answer on your question https://github.com/jenkinsci/pipeline-view-plugin/blob/master/README.md#jucies

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/YZAHSXYCfDY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/d46c322c-a897-43c7-b0ab-d3cc24d08eb6%40googlegroups.com.

[Michael Neale]

Oh neat! So we are using this instead of the "experimental" update center? 

(experimental means you release as normal, but as alpha, this seems to be quicker for more throw-away experiements)

[Sergei Egorov]

Hey Michael,

To be honest - I'm doing an experiment with Jucies, because I really want to have such way of distribution of plugins :) 

Maybe I'm too naive and users will deny using Jucies - this is fine as well, negative experiments are also experiments, right?

If I understand correctly, you want to play with it? So, since we're talking, what is your opinion about Jucies so far? 

BR,

Sergei

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/366ea45b-b474-4716-89d6-d89ca465e44e%40googlegroups.com.

[Michael Neale]

No - it's fine. I was suggesting it to someone else. I can easily try it myself, but thought it was in either experimental or the real update center. This is the first time I have heard of Juices and it sounds like a well worthwhile experiment to me - so lets see how it goes!