I'm trying to learn Jenkins Pipeline - pointers to good programmer documentation? Or is Jenkins World the place I should go????

[Carruth, Rusty]

I’ve read the “getting started with pipeline” pages (https://jenkins.io/doc/book/pipeline/getting-started/).

 

I’ve spent literally hours looking for something really helpful, in one place, for someone who wants to write CODE in the pipeline script.

 

I did just find https://jenkins.io/doc/book/pipeline/syntax/, which is a great step in the direction I was looking for.

 

However, an update.  I’ve been searching around and gotten a lot further, but I want more, quicker.

 

I should mention that I’ve been a software developer for many years (oh, what, 40 or so?  C, C++ primarily, but a little Java), I teach a class called “Linux Operating System” at a community college (have been for 3 years or so), I write scripts (bash, python, perl, etc) all the time, so I hope I have more than a small clue about ‘things’.

 

I note that Jenkins World is coming soon to SF.  Perhaps that’s the best place to learn/etc…

 

Here’s the deal.  I’m trying to Jenkins-ize our test infrastructure.  Right now it is all done via shell scripts and ssh.

 

However, it’s a monolith. You set your configuration for each test host (in a pretty big config file), start the run, and wait (up to more than a day! (sometimes multiple days)) for the test run to complete. When the test run completes, a script runs which generates a report, including of what tests ran, which of those that ran failed, and so forth (NOTE that just because a single test fails does NOT mean we want to kill the entire test run).

 

Oh, yes – I should mention that the test run involves MULTIPLE test hosts, each running basically independently of each other.  Once a test host finishes its test run, it is available for another test run.  Every day we build our system and that triggers a full test run, as defined by a config file (which defines which hosts to run, and the test config file to use for that host – each host can have a different test configuration. Indeed, it can have a completely different product being tested).

 

To do a test run on a host, we first run some scripts on the server which sets up the global environment (we monitor serial port activity and store the output in a log file in the test report directory, for example), then we set up stuff on the test host (which starts out running Linux) for the test to come.  Right now, MOST of our tests are run under FreeDOS (don’t ask – we currently have no choice in that matter), then when those tests finish the host reboots into Linux, runs any Linux-based tests, and then runs ‘finish’, which copies the logs to the server, creates the test report, and emails it to a specified group.

 

For one thing, we’d like to have better visibility into the progress of the test (even possibly with partial results), and the ability to stop the testing at any stage in the process fairly easily.

 

So, I’m working on re-doing it all with Jenkins driving things.

 

Right now I’ve got one pipeline which will use a config file to specify which test hosts (and configuration) to start running – its more or less done (reads config file, parses, does what it says).

 

I’ve got another pipeline which runs the ‘setup global environment’ script, and which I’ve just proven works fine, as far as it goes.  No scripts AFTER that have been implemented…

 

I’m working on implementing the rest of the thing (both the ‘actually start the run’ part that happens after the global environment setup, and all the different stages.  Right now I’ve got a pipeline which parses another config file and runs the scripts specified (and their command-line args).  It kind of works too, but I’m thinking that the command-line args are way too unfriendly for ‘normal’ users.

 

So, I’m halfway competent (which means at least 50% INcompetent!), but feel like if I knew a bit more I’d not rush down a path that I’ll have to redo later.

 

Are there any good documents to help me move more into the competent phase?  As I asked above, is Jenkins World the best place (bang for buck)?

 

Thanks!

 

Rusty

 

 

 

 

[Stephen Connolly]

On Wed 2 Aug 2017 at 00:24, Carruth, Rusty <Rusty....@smartm.com> wrote:

I’ve read the “getting started with pipeline” pages (https://jenkins.io/doc/book/pipeline/getting-started/).

 

I’ve spent literally hours looking for something really helpful, in one place, for someone who wants to write CODE in the pipeline script.

If you are trying tow write CODe in pipeline then IMHO you are doing it wrong.

Pipeline IMHO is the thin layer of orchestration glue that you layer on top at the end.

If I find myself writing code in pipeline that indicates to me that I have failed to factorise the build correctly.

Otoh I hear tales of people with 20k LoC pipelines (and shudder at the mess I believe they have made for themselves)

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/BY2PR0401MB09661B1CC7B181FFBF46F0129DB30%40BY2PR0401MB0966.namprd04.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

--

Sent from my phone

[Traiano Welcome]

+1 for this question.

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.

[bbyjenkns]

I'm struggling with the same thing - doc is very light or wants you to shell out for everything. I want to write groovy for everything, but it seems like simple things, such as using AntBuilder for it's file operation magic is very difficult if not impossible. 

[Stephen Connolly]

On 2 August 2017 at 06:10, bbyjenkns <mcj...@gmail.com> wrote:

I'm struggling with the same thing - doc is very light or wants you to shell out for everything. I want to write groovy for everything, but it seems like simple things, such as using AntBuilder for it's file operation magic is very difficult if not impossible. 

Pipeline is NOT groovy.

It is a CPS engine built on top of Groovy... it may look like Groovy, it may even sometimes walk and quack like Groovy, but your life will be infinitely better if you just accept that it is NOT Groovy.

Global Shared Libraries is where you go if you want to write idiomatic Groovy, and even there you can hit issues unless you truly understand the CPS magic and its full implications.

Use pipeline as a final orchestration glue layer and your life will be much easier

 

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d316f5f2-3620-4472-bc44-2aa7422c31da%40googlegroups.com.

[bbyjenkns]

Ok - understood. That does clear some things up. Where can I find more info on what is and is not allowed in global pipelines libraries? I'm writing one now, and have defaulted to using the 'sh' step and returning status codes to do most every file operation because what I try to do in base groovy just does not work (using AntBuilder to copy directories for example).

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

[Stephen Connolly]

On 2 August 2017 at 08:14, bbyjenkns <mcj...@gmail.com> wrote:

Ok - understood. That does clear some things up. Where can I find more info on what is and is not allowed in global pipelines libraries? I'm writing one now, and have defaulted to using the 'sh' step and returning status codes to do most every file operation because what I try to do in base groovy just does not work (using AntBuilder to copy directories for example).

FYI AntBuilder is not going to work because it is running on the master not on the build agent...

 

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/46ec8239-73d3-4ff7-9a7a-4e8705181d6b%40googlegroups.com.

[jer...@bodycad.com]

I totally agree with Stephen here, the pipeline script engine is a lock down groovy with many quirks and approval mess. Making it a painful experience. Do as much as possible into your build scripts (.vcxproj, makefile, shell script, python...). Just launch them and take light decision into pipeline. If you shell scripts are working, you may want to clean them to just launch the build and run sequence and let Jenkins pipeline deceide which node should be launch and do the tests on each platform/OS.

The declarative pipeline seem to make it easier to do things like this, but from what I have seen into this forum so far, many things ain't working so well yet, but it's definitely the future thing with blue ocean.

I for one do use jenkins pipeline (non declarative one) and it make only 243 lines  (with a lots of comments and even file headers) that does all the following:

1. checkout repos
2. checkout an helper repos for scripts (python, executable that does specific jobs)
3. build whole solution with msbuild
4. build unit tests
5. run the tests
6. extract results results
7. extract benchmarks data and graph them between build (into an HTML web graph)
   
8. Syntax checking for qmllint, Qml dir inspection, .vcxproj, .csproj 
9. Generate the Doxygen doc

Many of those things are done into python outside pipeline script, the pipeline only launch them (using either sh or bat) back to back as different stage.

[bbyjenkns]

FYI AntBuilder is not going to work because it is running on the master not on the build agent...

So, using java.nio.file.Files for example wouldn't work to copy files on any nodes other than master? How do you do any file operations inside a workspace on a slave otherwise? Do you have to shell out and cp everything? If that's the case, how do you properly handle errors etc?

[bbyjenkns]

I should mention that I'm specifically trying to perform file operations within a workspace using global pipeline libraries. I'm not following how a global pipeline library would even make sense if it can only run on master - am I missing something here?

[jer...@bodycad.com]

>> Do you have to shell out and cp everything? If that's the case, how do you properly handle errors etc?

The sh/bat have the returnStatus and returnStdout. Sadly no returnStdError, so you need to pipe and I suggest to use 'tee' to duplicate it if need such feature. 

https://jenkins.io/doc/pipeline/steps/workflow-durable-task-step/