Pipeline - stage, node

[Sharan Basappa]

Hello,

I am trying to adapt pipeline as a part of my project. I am starting with a simple example, though, to get my understanding correct.

However, after reading several white papers and even after being able to write simple examples, I am stuck with conceptual understanding of pipeline.

My first question is, what exactly is the use of stage. Looking at example it is clear that it is used to segregate distinct build phases but the question

is, whether it is inserted just to get stage based view in the build dashboard or there is more to it than this.

I have also seen stage being used to segregate like build, test and publish.

Does it mean that these are executed in sequential?

My second question is related to node. If I have 2 nodes, do they execute in parallel invariably?

Also, do different nodes create their own workspace?

Continuing with nodes, assume I have to do compile, run tests. Also, I can run two test suites in parallel (suite_1 and suite_2).

Now, I need to build first and only then I can run the 2 test suites.

How can I create nodes in this case?

Also, if each node creates its own workspace then how can node that run suite_1 and suite_2 access the compiled libraries

that is created by first node that compiles?

Thanks in advance ...

[Sharan Basappa]

can anyone be kind enough to respond to the query below of mine?

It would help me a lot

[Daniel Beck]

> On 21.01.2017, at 19:24, Sharan Basappa <sharan....@gmail.com> wrote:
>
> I am stuck with conceptual understanding of pipeline.

Have you read https://jenkins.io/doc/book/pipeline/ ? Most of what you're asking is being answered there.

[jer...@bodycad.com]

I may not have all answer here, but here a start.

stage serve no purpose except giving visual queue and split console data into the view as far as I know. It also split the time used to complete.

For the node, they are sequential unless you use the parallel instruction. So you probably need to build on a possible node first then parallel the unit test execution:

node('buildrequirement')
{
   // build instruction here
}
parallel
(
 platform1:
 {
 node('platform1requirements')
 {
 // run unit test on platform 1
 }
 },
 platform2:
 {
 node('platform2requirements')
 {
 // run unit test on platform 2
 }
 }
)

For better parallel example:

https://github.com/jenkinsci/pipeline-examples/blob/master/pipeline-examples/jobs-in-parallel/jobs_in_parallel.groovy

[Sharan Basappa]

I would just like to point out a note from pipeline paper I am reading. It is as follows,

Stages are usually the top most element of Pipeline syntax. Stages allow you to group your build step into its component parts. By default, multiple builds of the same pipeline can run concurrently. The stage element also allows you to control this concurrency:

The above seem to suggest that stage is necessary to control concurrency.

Also, one more follow-up question. If I were to build on multiple nodes then how would results from one node be available to the other node.

I assume node can be an independent machine with its own workspace.

[jer...@bodycad.com]

I'm not sure they actually do this much, but you could totally put a stage around your parallel. If we take a look at the diagram on

https://jenkins.io/doc/book/pipeline/

stage{
  parallel(['toto': node(){ ... }])  
}

seem like a valid way.

I for one have a node that contain multiple stage and it doesn't disturb it.

node
{
  stage('build')
  {
    ...
  }
  stage('unit testing')
  {
    ...
  }
}

I think you cannot declare a stage inside a parallel, it would mess up the stages. It's my understanding of them so far, they kind of just indicate a split between flow group.  Maybe the dev people from Jenkins or more advanced users can shed more light on this.

[Peter McNab]

I think the things missing from those docs are more explanation behind
some of the design choices, and what the benefits and drawbacks could be.

The mention of the major limitations of the pipeline scripts are either
not documented or buried in obscure comments in Jira issues. For
example, an explanation of the need for serialization, what methods are
whitelisted, the fact that you can't import a jar file, etc. I've spend
weeks fighting those issues, and perhaps if I understood the underlying
logic, use case, and design intention I'd not have wasted my time.

I keep seeing the same questions being asked on this list, on Stack
Overflow, and other places, but none of that seems to make it back to
the first docs someone will see when they're trying to use the pipeline.

Pete