[JIRA] (JENKINS-56286) Multibranch Pipeline + P4Plugin + Helix Library + Polling = Infinite Build Loop

4 views
Skip to first unread message

lystor@gmail.com (JIRA)

unread,
Mar 1, 2019, 5:26:01 AM3/1/19
to jenkinsc...@googlegroups.com
Mykola Ulianytskyi updated an issue
 
Jenkins / Bug JENKINS-56286
Multibranch Pipeline + P4Plugin + Helix Library + Polling = Infinite Build Loop
Change By: Mykola Ulianytskyi
Summary: Multibranch Pipeline + P4Plugin + Helix Library + Polling = Infinite Build Loop
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

lystor@gmail.com (JIRA)

unread,
Mar 1, 2019, 5:43:03 AM3/1/19
to jenkinsc...@googlegroups.com
Mykola Ulianytskyi updated an issue
Hello

  

 We have problem with Multibranch Pipeline + P4Plugin Modern SCM Helix Library + Polling.

P4Plugin always finds changes in some streams (Polling Log has line "Changes found" but has no lines "... found change: XXX") and triggers new build.

 Some The problem appears on all streams are affected, some are not with different delay in time .

All streams have the same Jenkinsfile and Jenkins Library.

All workspaces in Polling logs have unique syncID .

 ( Multibranch Pipeline was recreated several times and logs checked) .

  

 The problem appears after a while, and I am not able to reproduce STR and create minimal Jenkinsfile .

  

* The problem appears after after a while, and I don't see the cause of it. only if Modern SCM Helix Library configration is used in Jenkins Library!*

*Workaround to this bug: migrate to Legacy SCM >   Perforce Software > Manual Workspace*

 

 Next actions don't solve sometimes trigger the problem:
* Jenkins restart
* Job cancellation
* Manual Job start
 * Jenkins restart

Deleting and re-adding of stream in "Branch Sources > Helix Streams > Include streams" with deleting of child pipeline (for build history cleanup) temporary solve the build loop.

 *Library Configuration:*
* Name: <some name>
* Default version: now
* Load implicitly: False
* Allow default version to be overridden: False
* Include Library changes in job recent changes: False
* Retrieval method: Modern SCM
* Source Code Management: Helix Library
* Perforce Credentials: jenkins
* Library Depot Path:   //playground/jenkins-main/library/...

* Multibranch Pipeline Configuration: *
* Branch Sources > Helix Streams
* Perforce Credentials: jenkins
* Script Path: Tools/Jenkins/jobs/build/ Jenkinsfile
* Add property:  (Deprecated) Suppress automatic SCM triggering
* Disable: Discard old items
* Include streams:

  
 {code:java}
     // k02 playground / jenkins- main
 {code}
      * // k02 playground / dev jenkins - prototype
    main / library / k02 vars / dev-prototype-99
{code}
 

*scm-polling foo1 . log groovy *
{code:java}
 Started on Feb 26, 2019 10:29:00 AM
P4: Polling on: master with:jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579
P4Task: cleanup Client: jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579
 def call ( p4 ) :cmd:... p4 client -o jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579 {
p4 client -o jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579     echo ( p4 "foo1" ) :stop:4
(p4):cmd:... p4 info }
p4 info(p4):stop:5 {code}
(p4):cmd:... p4 info
p4 info(p4):stop:6
(p4):cmd:... p4 client -o jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579
p4 client -o jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579(p4):stop:7
(p4):cmd:... p4 client -i
p4 client -iClient jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579 saved.(p4):stop:8
(p4):cmd:...   View:
 *       // k02 playground / dev-prototype/Tools/Jenkins/library/... // jenkins- lib-df5e5c28-4afd-4be8-802a-ff6832e38579 main / ... Jenkinsfile*
(p4) {code : stop:P4 Task: establishing connection. java}
... server: perforce:1666 pipeline {
...     agent {
 node : k02-jenkins {
P4 Task: cleanup client: jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579
             label ( p4 "master" ) :cmd:... p4 client -o jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579
p4 client -o jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579(p4):stop:9         }
(p4):cmd:... p4 revert /var/jenkins_home/jobs/build/branches/dev-prototype/...     }
p4 revert /var/jenkins_home/jobs/build/branches/dev-prototype/.../var/jenkins_home/jobs/build/branches/dev-prototype/... - file(s) not opened on this client.(p4):stop:10
P4 Task: remove client: jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579     libraries {
(p4):cmd:... p4 client -d jenkins- lib -df5e5c28-4afd-4be8-802a-ff6832e38579
p4 client -d jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579Client jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579 deleted. ( p4 "foo" ) :stop:11
    }

P4: Polling on: master with:k02-dev-prototype--poll--master     options {
Found last change 1181055 on syncID k02-dev-prototype--poll--master
         disableConcurrentBuilds ( p4 ) :cmd:... p4 client -o k02-dev-prototype--poll--master
p4 client -o k02-dev-prototype--poll--master         skipDefaultCheckout ( p4 true ) :stop:4
(p4):cmd:... p4 info     }
p4 info(p4):stop:5
(p4):cmd:... p4 info     triggers {
p4 info         pollSCM ( p4 "* * * * *" ) :stop:6
(p4):cmd:... p4 client -o k02-dev-prototype--poll--master     }
p4 client -o k02-dev-prototype--poll--master(p4):stop:7
(p4):cmd:... p4 client -i     stages {
p4 client -iClient k02-dev-prototype--poll--master not changed.         stage ( p4 "Init" ) :stop:8 {
(p4):cmd:...   View:             steps {
      null null
(p4)                 checkout scm : stop:P4 Task: establishing connection.
... server: perforce (credential : 1666
... node: k02 'perforce - ticket- jenkins
P4: Polling with range: 1181055 ' , now
 populate: flushOnly ( p4) pin : cmd:... p4 changes -m20 //k02-dev-prototype--poll--master/...@1181055,now
p4 changes -m20 //k02-dev-prototype--poll--master/...@1181055,nowChange 1181055 on 2019/02/25 by yuriyp@k02-dev-prototype-art--kiepc2056 ' !U concept
 '
(p4) , quiet : stop:9
(p4 false ) :cmd:... p4 repos -C ,
p4 repos -C(p4)                      workspace : stop:10


P4: Polling on: master with:k02-dev-prototype--build-win64-sync--k02-bld-win-01
Found last change 1181055 on syncID k02-dev-prototype--build-win64-sync--k02-bld-win-01
 streamSpec ( p4) charset : cmd 'none', format : ... p4 client 'bar - o k02 - dev-prototype--build-win64-sync--k02-bld-win-01
p4 client -o k02-dev-prototype--build-win64-sync--k02-bld-win-01(p4) kiepc2006', pinHost : stop:4
(p4):cmd:... p4 info
p4 info(p4):stop:5
(p4):cmd:... p4 info
p4 info(p4):stop:6
(p4):cmd:... p4 client -o k02-dev-prototype--build-win64-sync--k02-bld-win-01
p4 client -o k02-dev-prototype--build-win64-sync--k02-bld-win-01(p4):stop:7
(p4):cmd:... p4 client -i
p4 client -iClient k02-dev-prototype--build-win64-sync--k02-bld-win-01 not changed.(p4):stop:8
(p4):cmd:...   View:
      null null
(p4):stop:P4 Task: establishing connection.
... server: perforce:1666
... node: k02-jenkins
P4: Polling with range: 1181055 true , 1181055
(p4) streamName : cmd:... p4 changes -m20 ' // k02-dev-prototype--build-win64-sync--k02-bld-win-01 playground / ...@1181055,1___
p4 changes jenkins - m20 //k02-dev-prototype--build-win64-sync--k02-bld-win-01/...@1181055,1181055Change 1181055 on 2019/02/25 by yuriyp@k02-dev-prototype-art--kiepc2056 main ' !U concept
'
(p4 ) :stop:9
(p4 ) :cmd:... p4 repos -C
p4 repos -CClient 'k02-dev-prototype--build-win64-sync--k02-bld-win-01' can only be used from host 'k02-bld-win-01'.                 echo ( p4 "hello" ) :stop:10


P4: Polling on: master with:k02-dev-prototype--build-linux-sync--k02-bld-win-02
Found last change 1181055 on syncID k02-dev-prototype--build-linux-sync--k02-bld-win-02
                 foo1 ( p4 ) :cmd:... p4 client -o k02-dev-prototype--build-linux-sync--k02-bld-win-02
p4 client -o k02-dev-prototype--build-linux-sync--k02-bld-win-02(p4):stop:4             }
(p4):cmd:... p4 info         }
p4 info(p4):stop:5     }
(p4):cmd:... p4 info }
p4 info(p4):stop:6 {code}
(p4):cmd:... p4 client *scm - o k02-dev-prototype--build-linux-sync--k02-bld-win-02
p4 client -o k02-dev-prototype--build-linux-sync--k02-bld-win-02(p4):stop:7
(p4):cmd: polling . .. p4 client -i log*
p4 client -iClient k02-dev-prototype--build-linux-sync--k02-bld-win-02 not changed.(p4) {code : stop:8 java}
(p4):cmd:...   View:
      null null
(p4):stop: P4 Task : establishing connection.
... server: perforce:1666
... node: k02-jenkins
P4: Polling on: master with range : 1181055,1181055
(p4):cmd:... p4 changes jenkins - m20 //k02 lib - dev df5e5c28 - prototype 4afd - 4be8 - build 802a - linux-sync--k02-bld-win-02/...@1181055,1___ ff6832e38579
p4 changes -m20 //k02-dev-prototype--build-linux-sync--k02-bld-win-02/ ... @1181055,1181055Change 1181055 on 2019/02/25 by yuriyp@k02-dev-prototype-art--kiepc2056 '!U concept
'
(p4):stop:9
(p4):cmd:... p4 repos -C
p4 repos -CClient 'k02-dev-prototype--build-linux-sync--k02-bld-win-02' can only be used from host 'k02-bld-win-02'.(p4):stop:10
Done. Took 0.22 sec

 Changes found
{code}
config.xml is attached.


 Any help would be really appreciated.

Thank you

lystor@gmail.com (JIRA)

unread,
Mar 1, 2019, 5:43:03 AM3/1/19
to jenkinsc...@googlegroups.com
Mykola Ulianytskyi updated an issue
Change By: Mykola Ulianytskyi
Attachment: config.xml

lystor@gmail.com (JIRA)

unread,
Mar 1, 2019, 5:43:03 AM3/1/19
to jenkinsc...@googlegroups.com
Mykola Ulianytskyi updated an issue
Change By: Mykola Ulianytskyi
Attachment: multibranch-config.xml

lystor@gmail.com (JIRA)

unread,
Mar 1, 2019, 5:43:03 AM3/1/19
to jenkinsc...@googlegroups.com
Mykola Ulianytskyi updated an issue
Change By: Mykola Ulianytskyi
Attachment: scm-polling.log

lystor@gmail.com (JIRA)

unread,
Mar 1, 2019, 5:43:03 AM3/1/19
to jenkinsc...@googlegroups.com
Mykola Ulianytskyi updated an issue
Change By: Mykola Ulianytskyi
Attachment: multibranch-state.xml

lystor@gmail.com (JIRA)

unread,
Mar 1, 2019, 6:03:02 AM3/1/19
to jenkinsc...@googlegroups.com
Mykola Ulianytskyi updated an issue
Change By: Mykola Ulianytskyi
Attachment: image-2019-03-01-13-01-53-711.png
Attachment: 01.png
Hello

We have problem with Multibranch Pipeline + Modern SCM Helix Library + Polling.


P4Plugin always finds changes in some streams (Polling Log has line "Changes found" but has no lines "... found change: XXX") and triggers new build.

The problem appears on all streams with different delay in time.


All streams have the same Jenkinsfile and Jenkins Library.

All workspaces in Polling logs have unique syncID (Multibranch Pipeline was recreated several times and logs checked).

The problem appears after a while, and I am not able to reproduce STR.

 

*The problem appears only if Modern SCM Helix Library configration is used in Jenkins Library!*

*Workaround to for this bug: migrate to Legacy SCM > Perforce Software > Manual Workspace*

 

Next actions sometimes trigger the problem:

* Manual Job start
* Jenkins restart

Deleting and re-adding of stream in "Branch Sources > Helix Streams > Include streams" with deleting of child pipeline (for build history cleanup) temporary solve the build loop.

*Library Configuration:*
* Name: <some name>
* Default version: now
* Load implicitly: False
* Allow default version to be overridden: False
* Include Library changes in job recent changes: False
* Retrieval method: Modern SCM
* Source Code Management: Helix Library
* Perforce Credentials: jenkins
* Library Depot Path: //playground/jenkins-main/library/...

*Multibranch Pipeline Configuration:*

* Branch Sources > Helix Streams
* Perforce Credentials: jenkins
* Script Path: Jenkinsfile

* Add property:  (Deprecated) Suppress automatic SCM triggering
* Disable: Discard old items
* Include streams:

{code:java}
//playground/jenkins-main
{code}
 *//playground/jenkins-main/library/vars/foo1.groovy*
{code:java}
def call() {
    echo("foo1")
}
{code}
*//playground/jenkins-main/Jenkinsfile*
{code:java}
pipeline {
    agent {
        node {
            label("master")
        }
    }

    libraries {
        lib("foo")
    }

    options {
        disableConcurrentBuilds()
        skipDefaultCheckout(true)
    }

    triggers {
        pollSCM("* * * * *")
    }

    stages {
        stage("Init") {
            steps {
                checkout scm: perforce(credential: 'perforce-ticket-jenkins', populate: flushOnly(pin: '', quiet: false),
                     workspace: streamSpec(charset: 'none', format: 'bar--kiepc2006', pinHost: true, streamName: '//playground/jenkins-main'))
                echo("hello")
                foo1()
            }
        }
    }
}
{code}
*scm-polling.log*
{code:java}

P4: Polling on: master with:jenkins-lib-df5e5c28-4afd-4be8-802a-ff6832e38579
...
Changes found
{code}
Any help would be really appreciated
 

I have spent some hours in IDE and found an interesting case:

!image-2019-03-01-13-01-53-711 . png|width=955,height=522!

 

If the problem occurs,
on each poll iteration the method {{TagAction.getLastChange(Run<?, ?> run, TaskListener listener, String syncID)}} *is always executed with the same syncID value*
but lastActions(run) returns the action[0] *with the different syncID value on each poll iteration*.

Result: {{TagAction.getLastChange}} returns empty list because the following expression is always false:
{code:java}
for (TagAction action : actions) {
   if (syncID.equals(action.getSyncID())) {
      ...{code}
{{Result: PerforceSCM.}}lookForChanges returns null because lastRefs is empty and a build triggers again.
{code:java}
List<P4Ref> lastRefs = TagAction.getLastChange(lastRun, listener, syncID);

if (lastRefs == null || lastRefs.isEmpty()) {
   // no previous build, return null.
   return null;
}

{code}
Please fix the bug.

 Thank you

lystor@gmail.com (JIRA)

unread,
Mar 1, 2019, 6:04:02 AM3/1/19
to jenkinsc...@googlegroups.com
Mykola Ulianytskyi updated an issue
Hello

We have problem with Multibranch Pipeline + Modern SCM Helix Library + Polling.

P4Plugin always finds changes in some streams (Polling Log has line "Changes found" but has no lines "... found change: XXX") and triggers new build.

The problem appears on all streams with different delay in time.

All streams have the same Jenkinsfile and Jenkins Library.

All workspaces in Polling logs have unique syncID (Multibranch Pipeline was recreated several times and logs checked).

The problem appears after a while, and I am not able to reproduce STR.

 

*The problem appears only if Modern SCM Helix Library configration is used in Jenkins Library!*

*Workaround for this bug: migrate to Legacy SCM > Perforce Software > Manual Workspace*
I have spent some hours in IDE and found an interesting case:

! image-2019-03- 01 -13-01-53-711 .png|width=955,height=522!


 

If the problem occurs,
on each poll iteration the method {{TagAction.getLastChange(Run<?, ?> run, TaskListener listener, String syncID)}} *is always executed with the same syncID value*
but lastActions(run) returns the action[0] *with the different syncID value on each poll iteration*.

Result: {{TagAction.getLastChange}} returns empty list because the following expression is always false:
{code:java}
for (TagAction action : actions) {
   if (syncID.equals(action.getSyncID())) {
      ...{code}
{{Result: PerforceSCM.}}lookForChanges returns null because lastRefs is empty and a build triggers again.
{code:java}
List<P4Ref> lastRefs = TagAction.getLastChange(lastRun, listener, syncID);

if (lastRefs == null || lastRefs.isEmpty()) {
   // no previous build, return null.
   return null;
}

{code}
Please fix the bug.

Thank you

lystor@gmail.com (JIRA)

unread,
Mar 1, 2019, 6:04:02 AM3/1/19
to jenkinsc...@googlegroups.com
Mykola Ulianytskyi updated an issue
Change By: Mykola Ulianytskyi
Attachment: image-2019-03-01-13-01-53-711.png

lystor@gmail.com (JIRA)

unread,
Mar 1, 2019, 6:12:02 AM3/1/19
to jenkinsc...@googlegroups.com
Mykola Ulianytskyi updated an issue
!01.png|width=955,height=522!


 

If the problem occurs,
on each poll iteration the method {{TagAction.getLastChange(Run<?, ?> run, TaskListener listener, String syncID)}} *is always executed with the same syncID value*
but lastActions(run) returns the action[0] *with the different syncID value on each poll iteration*.

Result: {{TagAction.getLastChange}} returns empty list because the following expression is always false:
{code:java}
for (TagAction action : actions) {
   if (syncID.equals(action.getSyncID())) {
      ...{code}
{{ Result: PerforceSCM. }} lookForChanges returns null because lastRefs is empty and a build triggers again . :
{code:java}
List<P4Ref> lastRefs = TagAction.getLastChange(lastRun, listener, syncID);

if (lastRefs == null || lastRefs.isEmpty()) {
   // no previous build, return null.
   return null;
}

{code}
Please fix the bug.

Thank you

lystor@gmail.com (JIRA)

unread,
Mar 1, 2019, 6:13:02 AM3/1/19
to jenkinsc...@googlegroups.com
Result: PerforceSCM.lookForChanges returns null because lastRefs is empty and a build triggers again:

{code:java}
List<P4Ref> lastRefs = TagAction.getLastChange(lastRun, listener, syncID);

if (lastRefs == null || lastRefs.isEmpty()) {
   // no previous build, return null.
   return null;
}

{code}
Please fix the bug.

Thank you

lystor@gmail.com (JIRA)

unread,
Mar 1, 2019, 10:48:03 AM3/1/19
to jenkinsc...@googlegroups.com
Mykola Ulianytskyi updated an issue
Hello

We have problem with Multibranch Pipeline + Modern SCM Helix Library + Polling.

P4Plugin always finds changes in some streams (Polling Log has line "Changes found" but has no lines "... found change: XXX") and triggers new build.

The problem appears on all streams with different delay in time.

All streams have the same Jenkinsfile and Jenkins Library.

All workspaces in Polling logs have unique syncID (Multibranch Pipeline was recreated several times and logs checked).

The problem appears after a while, and I am not able to reproduce STR.

*The problem appears only if Modern SCM Helix Library configration is used in Jenkins Library ! . *


*Workaround for this bug: migrate to Legacy SCM > Perforce Software > Manual Workspace*

Next actions sometimes trigger the problem:
* Manual Job start
* Jenkins restart

Deleting and re-adding of stream in "Branch Sources > Helix Streams > Include streams" with deleting of child pipeline (for build history cleanup) temporary solve the build loop but it re-appears after some time .

*Library Configuration:*
* Name: <some name>
* Default version: now
* Load implicitly: False
* Allow default version to be overridden: False
* Include Library changes in job recent changes: False
* Retrieval method: Modern SCM
* Source Code Management: Helix Library
* Perforce Credentials: jenkins
* Library Depot Path:   //playground/jenkins-main/library/...


* Multibranch Pipeline Configuration: Update #1 *
* Branch Sources > Helix Streams
* Perforce Credentials I have found one of possible steps to reproduce this bug : jenkins
* Script Path: Jenkinsfile
* Add property:  (Deprecated) Suppress automatic SCM triggering
* Disable: Discard old items
* Include streams:

{code:java}
1. Create //playground/jenkins-main stream
{code}
2. Submit   * //playground/jenkins-main/library/vars/foo1.groovy *
{code:java}
def call() {
    echo("foo1")
}
{code}
*  3. Submit //playground/jenkins-main/Jenkinsfile *
{code:java}
pipeline {
    agent {
        node {
            label ( "master" none )
        }
    }

     libraries {
        lib("foo")
    }

    options {
        buildDiscarder(logRotator(numToKeepStr: "10"))
         disableConcurrentBuilds()
         disableResume()
        quietPeriod(150)
        skipDefaultCheckout(true)
    }

    triggers {
        pollSCM("* * * * *")
    }

    stages {
        stage("Init") {
            agent {
                node {
                    label("master")
                }
            }
            steps {
                checkout scm: perforce(credential: ' " perforce-ticket-jenkins ' " , populate: flushOnly(pin: '' ""
, quiet: false),
                     workspace: streamSpec(charset: ' " none ' " , format: 'bar "test1 -- kiepc2006' ${NODE_NAME}" , pinHost: true, streamName: ' " //playground/ jenkins-main' ${BRANCH_NAME}" ))
                echo("hello")
                foo1()
                 sleep(80)
             }
        }
    }
}
{code}
 4. Run Jenkins container:
{code:java}
docker run --name=jenkins -p 8080:8080 jenkins/jenkins:2.150.3
{code}
5. Click button "Install suggested plugins"

 

6. Install p4 plugin (1.9.6)

7. Restart Jenkins

8. Add Perforce Credentials:
Jenkins > Credentials > Global > Add Credentials > Perforce Ticker Credential:
 * scm ID: perforce - ticket-jenkins
* P4Port: perforce:1666
* Username: jenkins
* Ticket: <censored>

9. Add Jenkins Library:
* Name: foo

* Default version: now
* Load implicitly: False
* Allow default version to be overridden: False
* Include Library changes in job recent changes: False
* Retrieval method: Modern SCM
* Source Code Management: Helix Library
* Perforce Credentials: jenkins
* Library Depot Path: //playground/jenkins-main/library/...

10. Add Multibranch Pipeline Job:
* Name: test1

* Branch Sources: Helix Streams
* Perforce Credentials: jenkins
* Include streams: //playground/jenkins-main
* Click "Save"

Result: test1/jenkins-main job will be automatically started.

11. Checkout and submit //playground/jenkins-main/library/vars/foo1.groovy
as soon as possible while test1/jenkins-main is running.

Result: infinite build loop.


Build interval between jobs - 3 minutes.
Polling starts each minute (* * * * *).

So polling is executed three times between builds and polling results are quite interesting:
# first polling in loop: Changes found => new build is scheduled (after 2 minutes due to quietPeriod)
# seconds polling in loop: No changes
# third polling in loop: No changes
# Job starts (quietPeriod expired)

And this loop repeats after each build (without any submits to perforce) . log

* Perforce Software Polling Log (first polling in loop)*
{code:java}
 Started on Mar 1, 2019 3:36:00 PM
 P4: Polling on: master with:jenkins-lib- df5e5c28 79530816 - 4afd 77d0 - 4be8 487e - 802a bb64 - ff6832e38579 61636b49a7bf
P4Task: cleanup Client: jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf
 ... p4 client -o jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
... p4 info +
... p4 info +
... p4 client -o jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
... p4 client -i +
... View: +

P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4 Task: cleanup client: jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf
... p4 client -o jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
... p4 revert /var/jenkins_home/jobs/test1/branches/jenkins-main/... +
P4 Task: remove client: jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf
... p4 client -d jenkins-lib-79530816-77d0-487e-bb64-61636b49a7bf +
P4: Polling on: master with:test1--master
Found last change 1183310 on syncID test1--master
... p4 client -o test1--master +
... p4 info +
... p4 info +
... p4 client -o test1--master +
... No change in client detected.

P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4: Polling with range: 1183310,now
... p4 changes -m20 //test1--master/...@1183310,now +
... p4 repos -C +
Done. Took 0.14 sec
 Changes found
{code}
*Perforce Software Polling Log (second polling in loop)*
{code:java}
Started on Mar 1, 2019 3:37:00 PM
P4: Polling on: master with:jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
Found last change 1183310 on syncID jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 info +
... p4 info +
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 client -i +
... View: +

P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4: Polling with range: 1183310,1183310
... p4 changes -m20 //jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6/...@1183310,1183___ +
... p4 repos -C +
P4Task: cleanup Client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 info +
... p4 info +
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 client -i +
... View: +

P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4 Task: cleanup client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 revert /var/jenkins_home/jobs/test1/branches/jenkins-main/... +
P4 Task: remove client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -d jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
P4: Polling on: master with:test1--master
Found last change 1183310 on syncID test1--master
... p4 client -o test1--master +
... p4 info +
... p4 info +
... p4 client -o test1--master +
... No change in client detected.

P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4: Polling with range: 1183310,now
... p4 changes -m20 //test1--master/...@1183310,now +
... p4 repos -C +
Done. Took 0.19 sec
No changes
{code}
*Perforce Software Polling Log (third polling in loop)*
{code:java}
Started on Mar 1, 2019 3:38:00 PM
P4: Polling on: master with:jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
Found last change 1183310 on syncID jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 info +
... p4 info +
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 client -i +
... View: +

P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4: Polling with range: 1183310,1183310
... p4 changes -m20 //jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6/...@1183310,1183___ +
... p4 repos -C +
P4Task: cleanup Client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 info +
... p4 info +
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 client -i +
... View: +

P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4 Task: cleanup client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -o jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
... p4 revert /var/jenkins_home/jobs/test1/branches/jenkins-main/... +
P4 Task: remove client: jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6
... p4 client -d jenkins-lib-cbe76982-a3c4-4e1f-abd3-6520b61e87e6 +
P4: Polling on: master with:test1--master
Found last change 1183310 on syncID test1--master
... p4 client -o test1--master +
... p4 info +
... p4 info +
... p4 client -o test1--master +
... No change in client detected.

P4 Task: establishing connection.
... server: perforce:1666
... node: fd9e6ec9c042
P4: Polling with range: 1183310,now
... p4 changes -m20 //test1--master/...@1183310,now +
... p4 repos -C +
Done. Took 0.2 sec
No changes
{code}
  

 Also I have spent some hours in IDE and found an interesting case:


!01.png|width=955,height=522!

 

If the problem occurs,
on each poll iteration the method {{TagAction.getLastChange(Run<?, ?> run, TaskListener listener, String syncID)}} *is always executed with the same syncID value*
but lastActions(run) returns the action[0] *with the different syncID value on each poll iteration*.

Result: {{TagAction.getLastChange}} returns empty list because the following expression is always false:
{code:java}
for (TagAction action : actions) {
   if (syncID.equals(action.getSyncID())) {
      ...{code}
Result: PerforceSCM.lookForChanges returns null because lastRefs is empty and a build triggers again:
{code:java}
List<P4Ref> lastRefs = TagAction.getLastChange(lastRun, listener, syncID);

if (lastRefs == null || lastRefs.isEmpty()) {
   // no previous build, return null.
   return null;
}

{code}
Please fix the bug.

Thank you

kwirth@perforce.com (JIRA)

unread,
Mar 5, 2019, 1:22:02 PM3/5/19
to jenkinsc...@googlegroups.com
Karl Wirth commented on Bug JENKINS-56286
 
Re: Multibranch Pipeline + P4Plugin + Helix Library + Polling = Infinite Build Loop

Hi Mykola Ulianytskyi, Thanks for the great investigation. I'm still trying to fully understand it. Two quick questions:

I beleive we assume the build history and config files will be accessible to the current job to figure out the SyncIDs and what the starting change to poll from should be.

(1) I'm seeing a lot of mention on this and other cases of docker images. Are these persistant master images that never get switch off or that have a fixed storage where on restart the full histoy is accessible?

(2) I'm assuming 'BuildDiscarder' deletes old build history. Is that correct? Is that something you use in all your jobs?

Thanks in advance.

dpn982@gmail.com (JIRA)

unread,
Mar 5, 2019, 2:01:05 PM3/5/19
to jenkinsc...@googlegroups.com

Hi Karl,

I am having this issue too. 

1.  We are using a persistent master container with a fixed storage that is mounted as a volume into the docker container.  We did recently upgrade our container (to 2.150.2) to a new version and the perforce plugin at the same time (to 1.9.6) but the build history is still the same as before the upgrade.

2.  We are using build discarder with all of our jobs.

 

I am not sure how Mykola has jenkins configured though.

Thanks,

Dave

lystor@gmail.com (JIRA)

unread,
Mar 6, 2019, 4:57:02 AM3/6/19
to jenkinsc...@googlegroups.com

Are these persistant master images that never get switch off or that have a fixed storage where on restart the full histoy is accessible?

If user don't specify host volume (/jenkins/volume/directory/on/host:/var/jenkins_home) or named volume (jenkins-volume:/var/jenkins_home) option Docker will create anonymous volume for /var/jenkins_home automatically.

See "VOLUME $JENKINS_HOME" on https://github.com/jenkinsci/docker/blob/master/Dockerfile

(2) I'm assuming 'BuildDiscarder' deletes old build history. Is that correct? Is that something you use in all your jobs?

Correct. Almost all Jenkins users use it to limit the build history.

lystor@gmail.com (JIRA)

unread,
Mar 6, 2019, 5:00:02 AM3/6/19
to jenkinsc...@googlegroups.com
Mykola Ulianytskyi edited a comment on Bug JENKINS-56286
{quote}Are these persistant master images that never get switch off or that have a fixed storage where on restart the full histoy is accessible?
{quote}
If user don't specify *host volume* (/jenkins/volume/directory/on/host:/var/jenkins_home) or *named volume* (jenkins-volume:/var/jenkins_home) option Docker will create *anonymous volume* for /var/jenkins_home automatically.

See "{{VOLUME $JENKINS_HOME"}} on [https://github.com/jenkinsci/docker/blob/master/Dockerfile]
In all cases Jenkins will have access to its /var/jenkins_home directory with full history after the restart.
 {quote}(2) I'm assuming 'BuildDiscarder' deletes old build history. Is that correct? Is that something you use in all your jobs?
{quote}

Correct. Almost all Jenkins users use it to limit the build history.

kwirth@perforce.com (JIRA)

unread,
Mar 8, 2019, 11:48:16 AM3/8/19
to jenkinsc...@googlegroups.com

Hi Mykola Ulianytskyi - Again thank you for all your work on this. Is the library being under a branch a normal situation for you or did you just do it force the bug? The problem is that if the library is under the branch it becomes included in two separate SyncIDs. The way we had assumed libraries would be used is that the library is external to the branch in some global location.

For example:

//build-dependencies/jenkins-main/library/vars/foo1.groovy
//playground/jenkins-main/Jenkinsfile
//playground/jenkins-main/src/f1.c
//playground/jenkins-main/src/f1.h

lystor@gmail.com (JIRA)

unread,
Mar 8, 2019, 11:55:01 AM3/8/19
to jenkinsc...@googlegroups.com

Hi Karl,

Is the library being under a branch a normal situation

The library and all Jenkins Pipelines belong to the same stream. The library contains common functions used by pipelines.

It is a normal practice and it works perfectly if used Legacy SCM > Perforce Software > Manual Workspace.

dpn982@gmail.com (JIRA)

unread,
Mar 8, 2019, 12:00:05 PM3/8/19
to jenkinsc...@googlegroups.com

Hi Karl,

our jenkins files are in the pipeline job in jenkins as text for most of our jobs.  Our pipeline code is in a different depot in Perforce from the code it builds.  Does this make sense?

Thanks,

Dave

kwirth@perforce.com (JIRA)

unread,
Mar 20, 2019, 1:28:02 PM3/20/19
to jenkinsc...@googlegroups.com

Hi Mykola Ulianytskyi - I have asked David to look for this pattern already, but can you please check if your builds go crazy after a delay in getting an executor? One pattern I have seen in David's case is that at the top of the console output there is more than one 'Started by an SCM change'. For example:

Started by an SCM change
Started by an SCM change 
Started by an SCM change

This seems to coincide with multiple polls occurring and finding the same change that needs to be built but being unable to run the build. For example:

 Poll 1 - Change 123 found - No executor available
 Poll 2 - Same change 123 found - No executor available
 Poll 3 - Same change 123 found - No executor available
 Poll 4 - Same change 123 found - No executor available
 Poll 5 - Executor becomes available - Runs build triggered by Poll 1
 Poll 6 - Executor available - Runs build triggered by Poll 2
 Poll 7 - Executor available - Runs build triggered by Poll 3
 Poll 8 - Does nothing (poll 4 has not triggered a build).

Do you see a similar pattern?

lystor@gmail.com (JIRA)

unread,
Mar 21, 2019, 12:19:02 PM3/21/19
to jenkinsc...@googlegroups.com

Do you see a similar pattern?

No. I have a different one.

You can repeat steps to reproduce from description of this bug and get required logs.

kwirth@perforce.com (JIRA)

unread,
Mar 21, 2019, 1:43:02 PM3/21/19
to jenkinsc...@googlegroups.com

Confirmed that the steps in the original description do cause infinite polling and do not cause multiple 'Started by an SCM change' however every invocation does start with a single 'Started by an SCM change'. Problem is reproducible with classic Perforce paths also.

Decreasing the sleep to 10 (sleep(10) ) in the above example and setting 5 second quiet period ( quietPeriod(5) ) stops the infinite builds running. Therefore overlapping polls is a factor and nested SyncIDs may not be relevant.

Passing to development.

kwirth@perforce.com (JIRA)

unread,
Mar 21, 2019, 1:44:02 PM3/21/19
to jenkinsc...@googlegroups.com
Karl Wirth assigned an issue to Unassigned
 
Change By: Karl Wirth
Assignee: Karl Wirth

kwirth@perforce.com (JIRA)

unread,
Mar 21, 2019, 1:44:03 PM3/21/19
to jenkinsc...@googlegroups.com
Karl Wirth updated an issue
Change By: Karl Wirth
Labels: P4_SUPPORT P4_A

kwirth@perforce.com (JIRA)

unread,
Apr 1, 2019, 5:48:01 AM4/1/19
to jenkinsc...@googlegroups.com
 
Re: Multibranch Pipeline + P4Plugin + Helix Library + Polling = Infinite Build Loop

FYI - David has highlighted the following changelist that may have had an impact here:

            https://swarm.workshop.perforce.com/changes/24591

Thoughts - Does this code also run for libraries? If not does the library code need a similar check?

lystor@gmail.com (JIRA)

unread,
Apr 1, 2019, 5:59:03 AM4/1/19
to jenkinsc...@googlegroups.com

If not does the library code need a similar check?

Libraries should have the same workaround because they usually are part of the same stream.

kwirth@perforce.com (JIRA)

unread,
Apr 1, 2019, 6:07:07 AM4/1/19
to jenkinsc...@googlegroups.com

Hi Mykola Ulianytskyi - Thanks. That is not the normal way we see users using it. Usually the libraries are in a different location from the source code. This is because the libraries tend to be  cross project and streams are often just one part of a project at a certain stage of development (for example game-engine at dev, main, qa, rel).

However this observation is just based on the support cases I have worked. There may be a large part of the user population that does put Jenkins libraries in the same stream as their source code.

lystor@gmail.com (JIRA)

unread,
Apr 1, 2019, 6:42:02 AM4/1/19
to jenkinsc...@googlegroups.com

Usually the libraries are in a different location from the source code.

1) Libraries in different location are used to share code between projects.

2) Libraries in the same stream are used to share common code between large pipelines in the same project. Pipelines can be different between streams so libraries belongs to streams for code consistency with pipelines.

pallen@perforce.com (JIRA)

unread,
Apr 4, 2019, 6:32:07 AM4/4/19
to jenkinsc...@googlegroups.com
Paul Allen assigned an issue to Paul Allen
 
Change By: Paul Allen
Assignee: Paul Allen

pallen@perforce.com (JIRA)

unread,
Apr 4, 2019, 6:32:10 AM4/4/19
to jenkinsc...@googlegroups.com
Paul Allen started work on Bug JENKINS-56286
 
Change By: Paul Allen
Status: Open In Progress

pallen@perforce.com (JIRA)

unread,
Jun 4, 2019, 9:29:01 AM6/4/19
to jenkinsc...@googlegroups.com
Paul Allen assigned an issue to Kevin Williamson
Change By: Paul Allen
Assignee: Paul Allen Kevin Williamson

pallen@perforce.com (JIRA)

unread,
Jun 4, 2019, 10:40:01 AM6/4/19
to jenkinsc...@googlegroups.com
Paul Allen updated an issue
Change By: Paul Allen
Labels: P4_A P4_DOC

pallen@perforce.com (JIRA)

unread,
Dec 5, 2019, 10:14:03 AM12/5/19
to jenkinsc...@googlegroups.com
Paul Allen updated an issue
Change By: Paul Allen
Priority: Blocker Major
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

kwirth@perforce.com (JIRA)

unread,
Dec 9, 2019, 11:39:04 AM12/9/19
to jenkinsc...@googlegroups.com
Karl Wirth assigned an issue to Karl Wirth
Change By: Karl Wirth
Assignee: Kevin Williamson Karl Wirth
Reply all
Reply to author
Forward
0 new messages