[JIRA] [pipeline-stage-view-plugin] (JENKINS-34644) Stage View Shows Excessively Long Full Run Time Figures

153 views
Skip to first unread message

svanoort@cloudbees.com (JIRA)

unread,
May 6, 2016, 10:36:02 AM5/6/16
to jenkinsc...@googlegroups.com
Sam Van Oort created an issue
 
Jenkins / Bug JENKINS-34644
Stage View Shows Excessively Long Full Run Time Figures
Issue Type: Bug Bug
Assignee: Sam Van Oort
Components: pipeline-stage-view-plugin
Created: 2016/May/06 2:35 PM
Labels: ux
Priority: Minor Minor
Reporter: Sam Van Oort

This one is hard to reproduce, but we have been seeing erratically large values for "Average full run time" in stage view, sometimes measured in years.

This may be caused by some specific failure in stages
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

svanoort@cloudbees.com (JIRA)

unread,
May 6, 2016, 10:40:02 AM5/6/16
to jenkinsc...@googlegroups.com

jsmith@privacystar.com (JIRA)

unread,
Jun 30, 2016, 3:16:01 PM6/30/16
to jenkinsc...@googlegroups.com
j smith commented on Bug JENKINS-34644

I am currently seeing this error occur on builds with under 10 runs and that have only been successful.
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

svanoort@cloudbees.com (JIRA)

unread,
Jun 30, 2016, 3:24:02 PM6/30/16
to jenkinsc...@googlegroups.com

j smith Since you have a case where it's actively occurring, any chance you could dump the /job/$JobName/wfapi/runs JSON content and attach it (along with what it's reporting for the full run time)?

I've had difficulty figuring out what triggers this bug, and having the data it's using for the computation would make it infinitely easier to trace the cause and solve it.

Thanks!

jenkins-ci.org@cowsgomoo.org (JIRA)

unread,
Jun 30, 2016, 3:33:02 PM6/30/16
to jenkinsc...@googlegroups.com

j smith asked me to add this - this is one branch in a pipeline multibranch job: 


[{
	"_links": {
		"self": {
			"href": "/job/p-c-b/job/PSAE-61/4/wfapi/describe"
		},
		"changesets": {
			"href": "/job/p-c-b/job/PSAE-61/4/wfapi/changesets"
		},
		"artifacts": {
			"href": "/job/p-c-b/job/PSAE-61/4/wfapi/artifacts"
		}
	},
	"id": "4",
	"name": "#4",
	"status": "SUCCESS",
	"startTimeMillis": 1467303127055,
	"endTimeMillis": 1467303874774,
	"durationMillis": 742082,
	"queueDurationMillis": 5637,
	"pauseDurationMillis": 0,
	"stages": [{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/4/execution/node/11/wfapi/describe"
			}
		},
		"id": "11",
		"name": "Clean workspace",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467303132692,
		"durationMillis": 1061,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/4/execution/node/14/wfapi/describe"
			}
		},
		"id": "14",
		"name": "Checkout source",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467303133753,
		"durationMillis": 1877,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/4/execution/node/17/wfapi/describe"
			}
		},
		"id": "17",
		"name": "Build App",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467303135630,
		"durationMillis": 189881,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/4/execution/node/27/wfapi/describe"
			}
		},
		"id": "27",
		"name": "Package Cookbook",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467303325511,
		"durationMillis": 23010,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/4/execution/node/36/wfapi/describe"
			}
		},
		"id": "36",
		"name": "Publish to S3",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467303348521,
		"durationMillis": 5144,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/4/execution/node/39/wfapi/describe"
			}
		},
		"id": "39",
		"name": "Archive site",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467303353665,
		"durationMillis": 2667,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/4/execution/node/45/wfapi/describe"
			}
		},
		"id": "45",
		"name": "Opsworks auto deployment",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467303356332,
		"durationMillis": 314976,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/4/execution/node/87/wfapi/describe"
			}
		},
		"id": "87",
		"name": "Test",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467303671308,
		"durationMillis": 1951,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/4/execution/node/93/wfapi/describe"
			}
		},
		"id": "93",
		"name": "Opsworks Tear Down",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467303673259,
		"durationMillis": 201515,
		"pauseDurationMillis": 0
	}]
},
{
	"_links": {
		"self": {
			"href": "/job/p-c-b/job/PSAE-61/3/wfapi/describe"
		},
		"changesets": {
			"href": "/job/p-c-b/job/PSAE-61/3/wfapi/changesets"
		},
		"artifacts": {
			"href": "/job/p-c-b/job/PSAE-61/3/wfapi/artifacts"
		}
	},
	"id": "3",
	"name": "#3",
	"status": "SUCCESS",
	"startTimeMillis": 1467299822359,
	"endTimeMillis": 1467300698534,
	"durationMillis": 864723,
	"queueDurationMillis": 11452,
	"pauseDurationMillis": 0,
	"stages": [{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/3/execution/node/11/wfapi/describe"
			}
		},
		"id": "11",
		"name": "Clean workspace",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467299833811,
		"durationMillis": 679,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/3/execution/node/14/wfapi/describe"
			}
		},
		"id": "14",
		"name": "Checkout source",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467299834490,
		"durationMillis": 5247,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/3/execution/node/17/wfapi/describe"
			}
		},
		"id": "17",
		"name": "Build App",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467299839737,
		"durationMillis": 370491,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/3/execution/node/27/wfapi/describe"
			}
		},
		"id": "27",
		"name": "Package Cookbook",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467300210228,
		"durationMillis": 22453,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/3/execution/node/36/wfapi/describe"
			}
		},
		"id": "36",
		"name": "Publish to S3",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467300232681,
		"durationMillis": 5288,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/3/execution/node/39/wfapi/describe"
			}
		},
		"id": "39",
		"name": "Archive site",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467300237969,
		"durationMillis": 2809,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/3/execution/node/45/wfapi/describe"
			}
		},
		"id": "45",
		"name": "Opsworks auto deployment",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467300240778,
		"durationMillis": 315573,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/3/execution/node/87/wfapi/describe"
			}
		},
		"id": "87",
		"name": "Test",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467300556351,
		"durationMillis": 1994,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/3/execution/node/93/wfapi/describe"
			}
		},
		"id": "93",
		"name": "Opsworks Tear Down",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467300558345,
		"durationMillis": 140189,
		"pauseDurationMillis": 0
	}]
},
{
	"_links": {
		"self": {
			"href": "/job/p-c-b/job/PSAE-61/2/wfapi/describe"
		},
		"changesets": {
			"href": "/job/p-c-b/job/PSAE-61/2/wfapi/changesets"
		},
		"artifacts": {
			"href": "/job/p-c-b/job/PSAE-61/2/wfapi/artifacts"
		}
	},
	"id": "2",
	"name": "#2",
	"status": "SUCCESS",
	"startTimeMillis": 1467299607985,
	"endTimeMillis": 1467300646688,
	"durationMillis": 1033210,
	"queueDurationMillis": 5493,
	"pauseDurationMillis": -1467300506456,
	"stages": [{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/2/execution/node/11/wfapi/describe"
			}
		},
		"id": "11",
		"name": "Clean workspace",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467299613478,
		"durationMillis": 1127,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/2/execution/node/14/wfapi/describe"
			}
		},
		"id": "14",
		"name": "Checkout source",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467299614605,
		"durationMillis": 1355,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/2/execution/node/17/wfapi/describe"
			}
		},
		"id": "17",
		"name": "Build App",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467299615960,
		"durationMillis": 516010,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/2/execution/node/28/wfapi/describe"
			}
		},
		"id": "28",
		"name": "Package Cookbook",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467300131970,
		"durationMillis": 33570,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/2/execution/node/37/wfapi/describe"
			}
		},
		"id": "37",
		"name": "Publish to S3",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467300165540,
		"durationMillis": 10096,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/2/execution/node/40/wfapi/describe"
			}
		},
		"id": "40",
		"name": "Archive site",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467300175636,
		"durationMillis": 5941,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/2/execution/node/46/wfapi/describe"
			}
		},
		"id": "46",
		"name": "Opsworks auto deployment",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467300181577,
		"durationMillis": 322850,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/2/execution/node/88/wfapi/describe"
			}
		},
		"id": "88",
		"name": "Test",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467300504427,
		"durationMillis": 2029,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/2/execution/node/94/wfapi/describe"
			}
		},
		"id": "94",
		"name": "Opsworks Tear Down",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467300506456,
		"durationMillis": -1467300506456,
		"pauseDurationMillis": -1467300506456
	}]
},
{
	"_links": {
		"self": {
			"href": "/job/p-c-b/job/PSAE-61/1/wfapi/describe"
		},
		"artifacts": {
			"href": "/job/p-c-b/job/PSAE-61/1/wfapi/artifacts"
		}
	},
	"id": "1",
	"name": "#1",
	"status": "SUCCESS",
	"startTimeMillis": 1467296834732,
	"endTimeMillis": 1467297918469,
	"durationMillis": 1069131,
	"queueDurationMillis": 14606,
	"pauseDurationMillis": 0,
	"stages": [{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/1/execution/node/11/wfapi/describe"
			}
		},
		"id": "11",
		"name": "Clean workspace",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467296849338,
		"durationMillis": 647,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/1/execution/node/14/wfapi/describe"
			}
		},
		"id": "14",
		"name": "Checkout source",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467296849985,
		"durationMillis": 4723,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/1/execution/node/17/wfapi/describe"
			}
		},
		"id": "17",
		"name": "Build App",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467296854708,
		"durationMillis": 573993,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/1/execution/node/28/wfapi/describe"
			}
		},
		"id": "28",
		"name": "Package Cookbook",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467297428701,
		"durationMillis": 22511,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/1/execution/node/37/wfapi/describe"
			}
		},
		"id": "37",
		"name": "Publish to S3",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467297451212,
		"durationMillis": 5798,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/1/execution/node/40/wfapi/describe"
			}
		},
		"id": "40",
		"name": "Archive site",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467297457010,
		"durationMillis": 2934,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/1/execution/node/46/wfapi/describe"
			}
		},
		"id": "46",
		"name": "Opsworks auto deployment",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467297459944,
		"durationMillis": 315086,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/1/execution/node/88/wfapi/describe"
			}
		},
		"id": "88",
		"name": "Test",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467297775030,
		"durationMillis": 2030,
		"pauseDurationMillis": 0
	},
	{
		"_links": {
			"self": {
				"href": "/job/p-c-b/job/PSAE-61/1/execution/node/94/wfapi/describe"
			}
		},
		"id": "94",
		"name": "Opsworks Tear Down",
		"execNode": "master",
		"status": "SUCCESS",
		"startTimeMillis": 1467297777060,
		"durationMillis": 141409,
		"pauseDurationMillis": 0
	}]
}]

jsmith@privacystar.com (JIRA)

unread,
Jun 30, 2016, 3:36:02 PM6/30/16
to jenkinsc...@googlegroups.com

It is currently displaying,

Average stage times:
(Average full run time: ~11y 230d)

svanoort@cloudbees.com (JIRA)

unread,
Jun 30, 2016, 3:53:01 PM6/30/16
to jenkinsc...@googlegroups.com

Thanks Joseph Smith – that shows where the problem is:

Run time for run #2 includes:

"queueDurationMillis": 5493,
    "pauseDurationMillis": -1467300506456,

and this is due to stage "Opsworks Tear Down" having the following timing info: 

"startTimeMillis": 1467300506456,
        "durationMillis": -1467300506456,
        "pauseDurationMillis": -1467300506456

This points at a bug in the stage timing computation in the REST API (not entirely surprising, it's deceptively complex, and the overall algorithm has several known issues that are being addressed with a new implementation).

One more question: is there anything odd about the "Opsworks Tear Down" stage in run #2? Does it include parallel structures?

jenkins-ci.org@cowsgomoo.org (JIRA)

unread,
Jun 30, 2016, 3:55:03 PM6/30/16
to jenkinsc...@googlegroups.com

It does not include parallel structures, but it is in a finally block.

svanoort@cloudbees.com (JIRA)

unread,
Jun 30, 2016, 4:30:01 PM6/30/16
to jenkinsc...@googlegroups.com

Note from investigation: last stage in the pipeline, negative pauseDuration could come from https://github.com/jenkinsci/pipeline-stage-view-plugin/blob/master/rest-api/src/main/java/com/cloudbees/workflow/rest/external/StageNodeExt.java#L122 if the durationMillis are negative.

startTime for that stage is positive... and duration exactly equal to it, but negative. That means endTime was 0... which only happens if there's no TimingAction set on the next stage node or the flow end node, AFAICT.

So the million dollar question to confirm the theory: in the last .xml file for jobs/$jobName/builds/2/workflow/ (something higher than 94), does the XML contain a TimingAction with a startTime?

I think solution here is probably to check if the TimingAction is or is not set here, and not do duration computation if it isn't set.

Coltrey Mather any chance you guys have that XML handy?

jenkins-ci.org@cowsgomoo.org (JIRA)

unread,
Jun 30, 2016, 6:30:25 PM6/30/16
to jenkinsc...@googlegroups.com

For complete-ishness, here's 100.xml:

<?xml version='1.0' encoding='UTF-8'?>
<org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage_-Tag plugin="workflow-support@2.1">
  <node class="org.jenkinsci.plugins.workflow.cps.nodes.StepAtomNode" plugin="workflow-cps@2.3">
    <parentIds>
      <string>99</string>
    </parentIds>
    <id>100</id>
    <descriptorId>org.jenkinsci.plugins.workflow.steps.SleepStep</descriptorId>
  </node>
  <actions>
    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
      <startTime>1467300509547</startTime>
    </org.jenkinsci.plugins.workflow.actions.TimingAction>
  </actions>
</org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage_-Tag>

I think the answer is yes:

[root@jenkins-master1 workflow]# grep TimingAction -A1 *
100.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
100.xml-      <startTime>1467300509547</startTime>
100.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
100.xml-  </actions>
--
101.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
101.xml-      <startTime>1467300569571</startTime>
101.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
101.xml-  </actions>
--
102.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
102.xml-      <startTime>1467300570898</startTime>
102.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
102.xml-  </actions>
--
103.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
103.xml-      <startTime>1467300570899</startTime>
103.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
103.xml-  </actions>
--
104.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
104.xml-      <startTime>1467300570900</startTime>
104.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
104.xml-  </actions>
--
105.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
105.xml-      <startTime>1467300630933</startTime>
105.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
105.xml-  </actions>
--
106.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
106.xml-      <startTime>1467300632244</startTime>
106.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
106.xml-  </actions>
--
107.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
107.xml-      <startTime>1467300632244</startTime>
107.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
107.xml-  </actions>
--
108.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
108.xml-      <startTime>1467300632245</startTime>
108.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
108.xml-  </actions>
--
109.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
109.xml-      <startTime>1467300634605</startTime>
109.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
109.xml-  </actions>
--
10.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
10.xml-      <startTime>1467299613477</startTime>
10.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
10.xml-  </actions>
--
110.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
110.xml-      <startTime>1467300634606</startTime>
110.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
110.xml-  </actions>
--
111.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
111.xml-      <startTime>1467300646581</startTime>
111.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
111.xml-  </actions>
--
112.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
112.xml-      <startTime>1467300646587</startTime>
112.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
112.xml-  </actions>
--
113.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
113.xml-      <startTime>1467300646624</startTime>
113.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
113.xml-  </actions>
--
114.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
114.xml-      <startTime>1467300646688</startTime>
114.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
114.xml-  </actions>
--
115.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
115.xml-      <startTime>1467300646745</startTime>
115.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
115.xml-  </actions>
--
11.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
11.xml-      <startTime>1467299613478</startTime>
11.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
11.xml-  </actions>
--
12.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
12.xml-      <startTime>1467299613479</startTime>
12.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
12.xml-  </actions>
--
13.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
13.xml-      <startTime>1467299614366</startTime>
13.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
13.xml-  </actions>
--
14.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
14.xml-      <startTime>1467299614605</startTime>
14.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
14.xml-  </actions>
--
15.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
15.xml-      <startTime>1467299614605</startTime>
15.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
15.xml-  </actions>
--
16.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
16.xml-      <startTime>1467299615321</startTime>
16.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
16.xml-  </actions>
--
17.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
17.xml-      <startTime>1467299615960</startTime>
17.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
17.xml-  </actions>
--
18.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
18.xml-      <startTime>1467299615961</startTime>
18.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
18.xml-  </actions>
--
19.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
19.xml-      <startTime>1467299615962</startTime>
19.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
19.xml-  </actions>
--
20.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
20.xml-      <startTime>1467299615962</startTime>
20.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
20.xml-  </actions>
--
21.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
21.xml-      <startTime>1467299616246</startTime>
21.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
21.xml-  </actions>
--
22.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
22.xml-      <startTime>1467299680467</startTime>
22.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
22.xml-  </actions>
--
23.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
23.xml-      <startTime>1467299951871</startTime>
23.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
23.xml-  </actions>
--
24.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
24.xml-      <startTime>1467300121319</startTime>
24.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
24.xml-  </actions>
--
25.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
25.xml-      <startTime>1467300121337</startTime>
25.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
25.xml-  </actions>
--
26.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
26.xml-      <startTime>1467300131656</startTime>
26.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
26.xml-  </actions>
--
27.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
27.xml-      <startTime>1467300131720</startTime>
27.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
27.xml-  </actions>
--
28.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
28.xml-      <startTime>1467300131970</startTime>
28.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
28.xml-  </actions>
--
29.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
29.xml-      <startTime>1467300132011</startTime>
29.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
29.xml-  </actions>
--
30.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
30.xml-      <startTime>1467300133006</startTime>
30.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
30.xml-  </actions>
--
31.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
31.xml-      <startTime>1467300133007</startTime>
31.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
31.xml-  </actions>
--
32.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
32.xml-      <startTime>1467300133040</startTime>
32.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
32.xml-  </actions>
--
33.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
33.xml-      <startTime>1467300133097</startTime>
33.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
33.xml-  </actions>
--
34.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
34.xml-      <startTime>1467300133098</startTime>
34.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
34.xml-  </actions>
--
35.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
35.xml-      <startTime>1467300165423</startTime>
35.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
35.xml-  </actions>
--
36.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
36.xml-      <startTime>1467300165464</startTime>
36.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
36.xml-  </actions>
--
37.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
37.xml-      <startTime>1467300165540</startTime>
37.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
37.xml-  </actions>
--
38.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
38.xml-      <startTime>1467300165541</startTime>
38.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
38.xml-  </actions>
--
39.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
39.xml-      <startTime>1467300175150</startTime>
39.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
39.xml-  </actions>
--
3.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
3.xml-      <startTime>1467299609797</startTime>
3.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
3.xml-    <org.jenkinsci.plugins.workflow.support.actions.WorkspaceActionImpl>
--
40.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
40.xml-      <startTime>1467300175636</startTime>
40.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
40.xml-  </actions>
--
41.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
41.xml-      <startTime>1467300175637</startTime>
41.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
41.xml-  </actions>
--
42.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
42.xml-      <startTime>1467300176708</startTime>
42.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
42.xml-  </actions>
--
43.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
43.xml-      <startTime>1467300176709</startTime>
43.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
43.xml-  </actions>
--
44.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
44.xml-      <startTime>1467300181575</startTime>
44.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
44.xml-  </actions>
--
45.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
45.xml-      <startTime>1467300181576</startTime>
45.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
45.xml-  </actions>
--
46.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
46.xml-      <startTime>1467300181577</startTime>
46.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
46.xml-  </actions>
--
47.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
47.xml-      <startTime>1467300181578</startTime>
47.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
47.xml-  </actions>
--
48.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
48.xml-      <startTime>1467300181594</startTime>
48.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
48.xml-  </actions>
--
49.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
49.xml-      <startTime>1467300181633</startTime>
49.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
49.xml-  </actions>
--
4.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
4.xml-      <startTime>1467299609817</startTime>
4.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
4.xml-  </actions>
--
50.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
50.xml-      <startTime>1467300181666</startTime>
50.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
50.xml-  </actions>
--
51.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
51.xml-      <startTime>1467300181894</startTime>
51.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
51.xml-  </actions>
--
52.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
52.xml-      <startTime>1467300181896</startTime>
52.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
52.xml-  </actions>
--
53.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
53.xml-      <startTime>1467300181896</startTime>
53.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
53.xml-  </actions>
--
54.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
54.xml-      <startTime>1467300182034</startTime>
54.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
54.xml-  </actions>
--
55.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
55.xml-      <startTime>1467300186932</startTime>
55.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
55.xml-  </actions>
--
56.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
56.xml-      <startTime>1467300186932</startTime>
56.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
56.xml-  </actions>
--
57.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
57.xml-      <startTime>1467300186933</startTime>
57.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
57.xml-  </actions>
--
58.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
58.xml-      <startTime>1467300186934</startTime>
58.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
58.xml-  </actions>
--
59.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
59.xml-      <startTime>1467300189912</startTime>
59.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
59.xml-  </actions>
--
5.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
5.xml-      <startTime>1467299611070</startTime>
5.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
5.xml-  </actions>
--
60.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
60.xml-      <startTime>1467300189946</startTime>
60.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
60.xml-  </actions>
--
61.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
61.xml-      <startTime>1467300189949</startTime>
61.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
61.xml-  </actions>
--
62.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
62.xml-      <startTime>1467300189950</startTime>
62.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
62.xml-  </actions>
--
63.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
63.xml-      <startTime>1467300192905</startTime>
63.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
63.xml-  </actions>
--
64.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
64.xml-      <startTime>1467300192905</startTime>
64.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
64.xml-  </actions>
--
65.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
65.xml-      <startTime>1467300192906</startTime>
65.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
65.xml-  </actions>
--
66.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
66.xml-      <startTime>1467300195222</startTime>
66.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
66.xml-  </actions>
--
67.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
67.xml-      <startTime>1467300197610</startTime>
67.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
67.xml-  </actions>
--
68.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
68.xml-      <startTime>1467300197611</startTime>
68.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
68.xml-  </actions>
--
69.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
69.xml-      <startTime>1467300197612</startTime>
69.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
69.xml-  </actions>
--
6.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
6.xml-      <startTime>1467299611594</startTime>
6.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
6.xml-  </actions>
--
70.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
70.xml-      <startTime>1467300257603</startTime>
70.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
70.xml-  </actions>
--
71.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
71.xml-      <startTime>1467300258913</startTime>
71.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
71.xml-  </actions>
--
72.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
72.xml-      <startTime>1467300258914</startTime>
72.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
72.xml-  </actions>
--
73.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
73.xml-      <startTime>1467300258923</startTime>
73.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
73.xml-  </actions>
--
74.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
74.xml-      <startTime>1467300318940</startTime>
74.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
74.xml-  </actions>
--
75.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
75.xml-      <startTime>1467300320289</startTime>
75.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
75.xml-  </actions>
--
76.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
76.xml-      <startTime>1467300320290</startTime>
76.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
76.xml-  </actions>
--
77.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
77.xml-      <startTime>1467300320291</startTime>
77.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
77.xml-  </actions>
--
78.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
78.xml-      <startTime>1467300380312</startTime>
78.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
78.xml-  </actions>
--
79.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
79.xml-      <startTime>1467300381731</startTime>
79.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
79.xml-  </actions>
--
7.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
7.xml-      <startTime>1467299612119</startTime>
7.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
7.xml-  </actions>
--
80.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
80.xml-      <startTime>1467300381732</startTime>
80.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
80.xml-  </actions>
--
81.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
81.xml-      <startTime>1467300381733</startTime>
81.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
81.xml-  </actions>
--
82.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
82.xml-      <startTime>1467300441750</startTime>
82.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
82.xml-  </actions>
--
83.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
83.xml-      <startTime>1467300443098</startTime>
83.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
83.xml-  </actions>
--
84.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
84.xml-      <startTime>1467300443099</startTime>
84.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
84.xml-  </actions>
--
85.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
85.xml-      <startTime>1467300443100</startTime>
85.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
85.xml-  </actions>
--
86.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
86.xml-      <startTime>1467300503109</startTime>
86.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
86.xml-  </actions>
--
87.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
87.xml-      <startTime>1467300504426</startTime>
87.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
87.xml-  </actions>
--
88.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
88.xml-      <startTime>1467300504427</startTime>
88.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
88.xml-  </actions>
--
89.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
89.xml-      <startTime>1467300504427</startTime>
89.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
89.xml-  </actions>
--
8.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
8.xml-      <startTime>1467299612646</startTime>
8.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
8.xml-  </actions>
--
90.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
90.xml-      <startTime>1467300505790</startTime>
90.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
90.xml-  </actions>
--
91.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
91.xml-      <startTime>1467300505791</startTime>
91.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
91.xml-  </actions>
--
92.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
92.xml-      <startTime>1467300506454</startTime>
92.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
92.xml-  </actions>
--
93.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
93.xml-      <startTime>1467300506455</startTime>
93.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
93.xml-  </actions>
--
94.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
94.xml-      <startTime>1467300506456</startTime>
94.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
94.xml-  </actions>
--
95.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
95.xml-      <startTime>1467300506457</startTime>
95.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
95.xml-  </actions>
--
96.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
96.xml-      <startTime>1467300506458</startTime>
96.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
96.xml-  </actions>
--
97.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
97.xml-      <startTime>1467300508191</startTime>
97.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
97.xml-  </actions>
--
98.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
98.xml-      <startTime>1467300509545</startTime>
98.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
98.xml-  </actions>
--
99.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
99.xml-      <startTime>1467300509546</startTime>
99.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
99.xml-  </actions>
--
9.xml:    <org.jenkinsci.plugins.workflow.actions.TimingAction plugin="workflow-api@2.1">
9.xml-      <startTime>1467299613476</startTime>
9.xml:    </org.jenkinsci.plugins.workflow.actions.TimingAction>
9.xml-  </actions>

svanoort@cloudbees.com (JIRA)

unread,
Jul 6, 2016, 12:12:01 AM7/6/16
to jenkinsc...@googlegroups.com

Root cause (at least in one of my systems): something bizarrely corrupt in the flownode/action storage itself, such that the FlowEndNode doesn't point to a valid start node or show as having timing actions. The following will show in a case triggering this (replace test-installation-wf with the job name of choice, and 29 with the build of choice). 

import org.jenkinsci.plugins.workflow.graph.BlockEndNode;
((BlockEndNode)(Jenkins.instance.getItem("test-installation-wf").getBuildByNumber(29).execution.currentHeads.get(0))).getStartNode();

Result: org.jenkinsci.plugins.workflow.graph.FlowStartNode[id=null]

... and the FlowEndNode does NOT have any actions attached:

((BlockEndNode)(Jenkins.instance.getItem("test-installation-wf").getBuildByNumber(29).execution.currentHeads.get(0))).actions

When trying to fetch a node after the point at which it occurs (publicly visible API here), the above causes an NPE:

http://dynamic.codeablereason.com/jenkins/job/test-installation-wf/29/execution/node/22/wfapi/describe

Caused by: java.lang.NullPointerException
at org.jenkinsci.plugins.workflow.graph.FlowNode.getActions(FlowNode.java:259)
at hudson.model.Actionable.getAllActions(Actionable.java:94)
at hudson.model.Actionable.getAction(Actionable.java:165)
at org.jenkinsci.plugins.workflow.cps.nodes.StepStartNode.isBody(StepStartNode.java:61)
at org.jenkinsci.plugins.workflow.cps.nodes.StepEndNode.getTypeDisplayName(StepEndNode.java:35)
at org.jenkinsci.plugins.workflow.graph.FlowNode.getDisplayName(FlowNode.java:169)
at com.cloudbees.workflow.rest.external.FlowNodeExt.addBasicNodeData(FlowNodeExt.java:168)
at com.cloudbees.workflow.rest.external.FlowNodeExt.addBasicNodeData(FlowNodeExt.java:197)
at com.cloudbees.workflow.rest.external.FlowNodeExt.create(FlowNodeExt.java:144)
at com.cloudbees.workflow.rest.endpoints.flownode.Describe.get(Describe.java:71)
at com.cloudbees.workflow.rest.endpoints.FlowNodeAPI.doDescribe(FlowNodeAPI.java:56)
... 93 more

And this lovely nastiness results (22 is the flow end node there):

Jenkins.instance.getItem("test-installation-wf").getBuildByNumber(29).execution.getNode("22").actions;

Describe result:
http://dynamic.codeablereason.com/jenkins/job/test-installation-wf/29/wfapi/describe

{"_links":{"self":{"href":"/jenkins/job/test-installation-wf/29/wfapi/describe"}},"id":"29","name":"#29","status":"SUCCESS","startTimeMillis":1450803921198,"endTimeMillis":1450803923964,"durationMillis":2667,"queueDurationMillis":99,"pauseDurationMillis":-1450803921297,"stages":[{"_links":{"self":{"href":"/jenkins/job/test-installation-wf/29/execution/node/5/wfapi/describe"}},"id":"5","name":"Build the sudo images for installation","execNode":"master","status":"SUCCESS","startTimeMillis":1450803921297,"durationMillis":-1450803921297,"pauseDurationMillis":-1450803921297}]}

*And when we probe by groovy console, lo and behold, both of these show no action... so no TimingInfo *

Jenkins.instance.getItem("test-installation-wf").getBuildByNumber(29).execution.getNode("22").getActions()
Jenkins.instance.getItem("test-installation-wf").getBuildByNumber(29).execution.currentHeads.get(0).getActions()

Thus we get 0 for the endTime (return value if node.getAction(TimingAction.class) == null).

svanoort@cloudbees.com (JIRA)

unread,
Jul 6, 2016, 12:36:02 AM7/6/16
to jenkinsc...@googlegroups.com

Coltrey Mather and Joseph Smith Hopefully the final one (this reproduces the problem for me consistently) – if you run the following in the script console for broken builds, does it yield an empty result 

Result: []

Script:

Jenkins.instance.getItem('myJobName').getBuildByNumber(myBuildNumber).execution.currentHeads.get(0))).actions

jenkins-ci.org@cowsgomoo.org (JIRA)

unread,
Jul 6, 2016, 9:56:03 AM7/6/16
to jenkinsc...@googlegroups.com

Sam Van Oort

This is a multibranch pipeline, so maybe it's different ... but this just gives me an NPE...I've verified that this is a build number with a negative durationMillis and pauseDurationMillis

Jenkins.instance.getItem('p-j-d/master').getBuildByNumber(25).execution.currentHeads.get(0).actions

java.lang.NullPointerException: Cannot invoke method getBuildByNumber() on null object
	at org.codehaus.groovy.runtime.NullObject.invokeMethod(NullObject.java:91)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.NullCallSite.call(NullCallSite.java:35)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at Script1.run(Script1.groovy:1)
	at groovy.lang.GroovyShell.evaluate(GroovyShell.java:585)
	at groovy.lang.GroovyShell.evaluate(GroovyShell.java:623)
	at groovy.lang.GroovyShell.evaluate(GroovyShell.java:594)
	at hudson.util.RemotingDiagnostics$Script.call(RemotingDiagnostics.java:142)
	at hudson.util.RemotingDiagnostics$Script.call(RemotingDiagnostics.java:114)
	at hudson.remoting.LocalChannel.call(LocalChannel.java:45)
	at hudson.util.RemotingDiagnostics.executeGroovy(RemotingDiagnostics.java:111)
	at jenkins.model.Jenkins._doScript(Jenkins.java:4043)
	at jenkins.model.Jenkins.doScript(Jenkins.java:4015)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:324)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:167)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:100)
	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
	at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:49)
	at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:44)
	at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:106)
	at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:44)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:499)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

svanoort@cloudbees.com (JIRA)

unread,
Jul 11, 2016, 10:06:03 AM7/11/16
to jenkinsc...@googlegroups.com

I have a workaround that resolves this issue now (in review).

jglick@cloudbees.com (JIRA)

unread,
Jul 11, 2016, 10:41:01 AM7/11/16
to jenkinsc...@googlegroups.com
Jesse Glick started work on Bug JENKINS-34644
 
Change By: Jesse Glick
Status: Open In Progress

svanoort@cloudbees.com (JIRA)

unread,
Jul 13, 2016, 10:31:02 AM7/13/16
to jenkinsc...@googlegroups.com
Sam Van Oort resolved as Fixed
 

Fix for this is merged, pending next release.
Change By: Sam Van Oort
Status: In Progress Resolved
Resolution: Fixed

svanoort@cloudbees.com (JIRA)

unread,
Jul 13, 2016, 12:38:01 PM7/13/16
to jenkinsc...@googlegroups.com
Sam Van Oort updated an issue
Change By: Sam Van Oort

svanoort@cloudbees.com (JIRA)

unread,
Jul 18, 2016, 6:11:05 PM7/18/16
to jenkinsc...@googlegroups.com
Sam Van Oort closed an issue as Fixed
 

Workaround released.

Change By: Sam Van Oort
Status: Resolved Closed

jenkins-ci.org@cowsgomoo.org (JIRA)

unread,
Jul 28, 2016, 11:33:01 AM7/28/16
to jenkinsc...@googlegroups.com
Coltrey Mather commented on Bug JENKINS-34644
 
Re: Stage View Shows Excessively Long Full Run Time Figures

Sam, does it require a build for the stage view avg time to update? I'm running 2.6 now and still see a branch with a ~2y avg build time, but I haven't build since the new version was installed.
Reply all
Reply to author
Forward
0 new messages