One-Shot Executors

[nicolas de loof]

Hi folks,

Yoann and I have extracted from our docker-slaves hack-ish pet project a stable sub-component so a plugin can manage slaves that are dedicated to a Build, not relying on Cloud API, and get them tied to a Build. i.e Slave and Build share a common lifecycle. If Slave fails to start, Build will fail as well. If slave is slow to start (maybe pulling a huge docker image), build log will report the progress.

etc.

design and use-case : https://wiki.jenkins-ci.org/display/JENKINS/One-Shot+Executor

code : https://github.com/jenkinsci/one-shot-executor-plugin

the current code base do rely on hacks, our goal is to demonstrate this use case (can be tested reusing sample) so we can get the adequate hooks introduced in jenkins-core and later re-implement same API on a cleaner basis.

[nicolas de loof]

Dominik edited the wiki page to fix a typo and now it is not available as https://wiki.jenkins-ci.org/display/JENKINS/One-Shot+Executor anymore, but still there https://wiki.jenkins-ci.org/pages/viewpage.action?pageId=96075993

Can a Confluence expert can explain this ?

[Kanstantsin Shautsou]

This is called cached. Everybody who uses jenkins jira hits this issue always.

[nicolas de loof]

There was actually an extra + sign on URL that broke the link

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/f5935828-61d5-4afa-afab-9090deacb19e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kanstantsin Shautsou]

Cloud delays in theory should be possible to avoid by:

1) kicking suggestReview. I wrote plugin but had no ability to test on huge installations https://github.com/KostyaSha/faster-node-provision-plugin

2) increasing one of coefficients for provisioner(or statistics collector, forgot classname) (jglick is using it in demo's and i'm using in my installations).

AFAIK the main jenkins pain is the Queue locks. How this plugin will deal with locks? 

[Baptiste Mathus]

2016-03-04 21:14 GMT+01:00 Kanstantsin Shautsou <kanstan...@gmail.com>:

Cloud delays in theory should be possible to avoid by:

1) kicking suggestReview. I wrote plugin but had no ability to test on huge installations https://github.com/KostyaSha/faster-node-provision-plugin

2) increasing one of coefficients for provisioner(or statistics collector, forgot classname) (jglick is using it in demo's and i'm using in my installations).

NodeProvisionner?

Btw, that page might be related IIUC what you talk about. There're many seemingly (didn't test yet) useful infos about those coeffs and what they do:

https://cloudbees.zendesk.com/hc/en-us/articles/204690520-Why-do-slaves-show-as-suspended-while-jobs-wait-in-the-queue-

Btw, IIUC, this page might explain why I'm seeing my slave/agent docker containers being provisionned in between 5 seconds and roughly 6 minutes...

AFAIK the main jenkins pain is the Queue locks. How this plugin will deal with locks? 

2016-03-04 12:08 GMT+01:00 nicolas de loof <nicolas...@gmail.com>:

Hi folks,

Yoann and I have extracted from our docker-slaves hack-ish pet project a stable sub-component so a plugin can manage slaves that are dedicated to a Build, not relying on Cloud API, and get them tied to a Build. i.e Slave and Build share a common lifecycle. If Slave fails to start, Build will fail as well. If slave is slow to start (maybe pulling a huge docker image), build log will report the progress.

etc.

design and use-case : https://wiki.jenkins-ci.org/display/JENKINS/One-Shot+Executor

code : https://github.com/jenkinsci/one-shot-executor-plugin

the current code base do rely on hacks, our goal is to demonstrate this use case (can be tested reusing sample) so we can get the adequate hooks introduced in jenkins-core and later re-implement same API on a cleaner basis.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/f5935828-61d5-4afa-afab-9090deacb19e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/d65c0d1f-e298-4e08-b5f9-15f883b55f12%40googlegroups.com.

[Kanstantsin Shautsou]

I made my plugin after @stephenc comments and seems that his work somebody finally tried to expose https://github.com/jenkinsci/mansion-cloud-plugin/pull/9 According to stephenc comments such logic should be generic and would make sense for all cloud plugins, that’s why i made separate plugin.

@batmat if you are using docker-plugin then it may have synchronisation or logic issues. 

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/WV6sPmIh-tk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANWgJS7kb2RkXzgQPrv7SS%2BCJVsV79q17sHsVE6pzYXrCh3QHw%40mail.gmail.com.

[nicolas de loof]

2016-03-04 21:14 GMT+01:00 Kanstantsin Shautsou <kanstan...@gmail.com>:

Cloud delays in theory should be possible to avoid by:

1) kicking suggestReview. I wrote plugin but had no ability to test on huge installations https://github.com/KostyaSha/faster-node-provision-plugin

2) increasing one of coefficients for provisioner(or statistics collector, forgot classname) (jglick is using it in demo's and i'm using in my installations).

Cloud API delay is not the main reason I'm working on this. For sure I preferred to bypass it vs working on such tweaks, which probably would benefit many Cloud slaves, but that's not my topic.

a One-Shot executor is tied to a Run, is defined by a Run. It's a paradigm change, not just a workaround for some performance issues. 

 

AFAIK the main jenkins pain is the Queue locks. How this plugin will deal with locks? 

2016-03-04 12:08 GMT+01:00 nicolas de loof <nicolas...@gmail.com>:

Hi folks,

Yoann and I have extracted from our docker-slaves hack-ish pet project a stable sub-component so a plugin can manage slaves that are dedicated to a Build, not relying on Cloud API, and get them tied to a Build. i.e Slave and Build share a common lifecycle. If Slave fails to start, Build will fail as well. If slave is slow to start (maybe pulling a huge docker image), build log will report the progress.

etc.

design and use-case : https://wiki.jenkins-ci.org/display/JENKINS/One-Shot+Executor

code : https://github.com/jenkinsci/one-shot-executor-plugin

the current code base do rely on hacks, our goal is to demonstrate this use case (can be tested reusing sample) so we can get the adequate hooks introduced in jenkins-core and later re-implement same API on a cleaner basis.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/f5935828-61d5-4afa-afab-9090deacb19e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/d65c0d1f-e298-4e08-b5f9-15f883b55f12%40googlegroups.com.

[nicolas de loof]

2016-03-05 7:36 GMT+01:00 Kanstantsin Shautsou <kanstan...@gmail.com>:

I made my plugin after @stephenc comments and seems that his work somebody finally tried to expose https://github.com/jenkinsci/mansion-cloud-plugin/pull/9 According to stephenc comments such logic should be generic and would make sense for all cloud plugins, that’s why i made separate plugin.

@batmat if you are using docker-plugin then it may have synchronisation or logic issues. 

@KostyaSha is this something you're trying to address with you "yet-another-docker-plugin" ? Are you working on both plugins ?

 

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/F0D26341-CD48-4C8C-BBF2-8F69CF060B12%40gmail.com.

[Kanstantsin Shautsou]

On Mar 5, 2016, at 11:15, nicolas de loof <nicolas...@gmail.com> wrote:

2016-03-05 7:36 GMT+01:00 Kanstantsin Shautsou <kanstan...@gmail.com>:

I made my plugin after @stephenc comments and seems that his work somebody finally tried to expose https://github.com/jenkinsci/mansion-cloud-plugin/pull/9 According to stephenc comments such logic should be generic and would make sense for all cloud plugins, that’s why i made separate plugin.

@batmat if you are using docker-plugin then it may have synchronisation or logic issues. 

@KostyaSha is this something you're trying to address with you "yet-another-docker-plugin" ? Are you working on both plugins ?

I’m not maintaining docker-plugin anymore (reasons described in readme). I had good experience with migrating static ssh slaves to docker-plugin, but it had not enough stability.  In yad-plugin i just continue the work and want get stable and faster provisioning for standard Cloud slaves (for integration tests invented remote run and configuration of master instance and started working on jmeter plugin for doing performance tests). Delays mostly not a cloud plugin issue, so experimental “faster” i placed in separate plugin. In yad-plugin and docker-java library i almost resolved all related issues and hope it will work good, stable and covered with ITs.

For future I have some experimental ideas for DSLs, but they may duplicate existing features with the difference that i will use docker-java. Even if nobody will use it, i would have reference implementation for docker-java where i’m co-maintainer. For example it should be possible to expose globally configured docker connection into JobProperty configuration (that of course will use QTD for hacks), but that would raise questions about limits and security. Everything depends on free time and mood :)

yad-plugin -  is released into UC (it my experiment to host it under personal account).

faster-node-provision-plugin - isn’t released as i had no final conclusion on it’s stability (maybe it would require to substitute the whole provisioning strategy). If stephenc will finally review and bless then it would make sense to fork and release it.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANMVJznxr4jD_hu8JYTS%3DEa%2B84Y6uVA0mr10XSTe76%2BHxcmDmA%40mail.gmail.com.

[Kanstantsin Shautsou]

`You've been removed from the Jenkins team docker-plugin Developers`

Thanks to anonymous person, removing person from team would be a great benefit to this plugin! ;)

That a bit offtop, please continue with the initial topic. 

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/d65c0d1f-e298-4e08-b5f9-15f883b55f12%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

-- 

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/WV6sPmIh-tk/unsubscribe.

To unsubscribe from this group and all its topics, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANWgJS7kb2RkXzgQPrv7SS%2BCJVsV79q17sHsVE6pzYXrCh3QHw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/F0D26341-CD48-4C8C-BBF2-8F69CF060B12%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/WV6sPmIh-tk/unsubscribe.

To unsubscribe from this group and all its topics, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

[nicolas de loof]

*you* just said you aren't working anymore on this plugin. So why complain ?

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/d65c0d1f-e298-4e08-b5f9-15f883b55f12%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

-- 

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/WV6sPmIh-tk/unsubscribe.

To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANWgJS7kb2RkXzgQPrv7SS%2BCJVsV79q17sHsVE6pzYXrCh3QHw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/F0D26341-CD48-4C8C-BBF2-8F69CF060B12%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/WV6sPmIh-tk/unsubscribe.

To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANMVJznxr4jD_hu8JYTS%3DEa%2B84Y6uVA0mr10XSTe76%2BHxcmDmA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/c027feb0-3b86-4dfd-bd83-7dafc8c7d743%40googlegroups.com.

[Kanstantsin Shautsou]

Was it you? I still answering in PRs and issues. @magnayan is also not actively maintaining it, would you also remove it? If it was you (the person that never worked on this plugin) then i would be glad to see the list of plugins what you maintain and ensure that you also removed from all unrelated teams. Would it be fair? Either i would request descriptions to the Jenkins Board. 

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANMVJznqSHnpf0H-hZA2fiqJ_mPNbR0d4b2nW5uGUefvH%2B9bew%40mail.gmail.com.

[Kanstantsin Shautsou]

Ok, offtopic should die, i sent email to board, i will get explanations and description from board.

For cloud faster provisioning stephenc replied that will look at possible ways on Monday.

[nicolas de loof]

Yes, was me.

I never will get used with your "style". You claim you don't work on a plugin, you even create another one, and now complain you've been removed from the plugin admin team ?

Feel free to report this "abuse" to board if this is such a disturbance for you, I'm sure someone will find the IRC bot command to give you back access to this repo.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/42AEF0EC-8E20-40A1-8B94-0F091E895726%40gmail.com.

[Suckow, Thomas J]

I like that this moves the provisioning to the build log.

I do agree that certain issues should fail immediately (image not found). Certain other issues should perform exponential backoff (Cloud infrastructure down). Provisioning limits could be annoying though, would be interesting if they could be left in the queue until Jenkins side provisioning limits are not violated. I am not sure how to handle an environment like Kubernetes though where other entities may be utilizing resources and you have to "share".

You mention using labels to pick the slave. I'm wondering if it would be feasible and worthwhile to make such a plugin generic to be the middle layer for the Jenkins hooks to the cloud specific implementation (Docker, Kubernetes, AWS).

It could also handle the logic of some users wanting to configure slaves on a per job basis. Would be interesting if could also be integrated into cloudbees folder level. If the later could work then I wouldn't need to run my own Jenkins install at work for using containers and instead could use the company cloudbees Jenkins.

-

Thomas

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANMVJznmnydODNX%2BYXmh0ujnJXsoZWcPT%2BpJCWwdR0_wUpfDTQ%40mail.gmail.com.

[Kanstantsin Shautsou]

On Mar 9, 2016, at 19:52, Suckow, Thomas J <Thomas...@pnnl.gov> wrote:

I like that this moves the provisioning to the build log.

I do agree that certain issues should fail immediately (image not found). Certain other issues should perform exponential backoff (Cloud infrastructure down). Provisioning limits could be annoying though, would be interesting if they could be left in the queue until Jenkins side provisioning limits are not violated. I am not sure how to handle an environment like Kubernetes though where other entities may be utilizing resources and you have to "share".

You mention using labels to pick the slave. I'm wondering if it would be feasible and worthwhile to make such a plugin generic to be the middle layer for the Jenkins hooks to the cloud specific implementation (Docker, Kubernetes, AWS).

It could also handle the logic of some users wanting to configure slaves on a per job basis. Would be interesting if could also be integrated into cloudbees folder level. If the later could work then I wouldn't need to run my own Jenkins install at work for using containers and instead could use the company cloudbees Jenkins.

I guess folder has some FolderProperty that you cat get from job.getParent(), that’s all.

-

Thomas

From: <jenkin...@googlegroups.com> on behalf of nicolas de loof <nicolas...@gmail.com>
Reply-To: "jenkin...@googlegroups.com" <jenkin...@googlegroups.com>
Date: Friday, March 4, 2016 at 3:08 AM
To: "jenkin...@googlegroups.com" <jenkin...@googlegroups.com>
Subject: One-Shot Executors

Hi folks,

Yoann and I have extracted from our docker-slaves hack-ish pet project a stable sub-component so a plugin can manage slaves that are dedicated to a Build, not relying on Cloud API, and get them tied to a Build. i.e Slave and Build share a common lifecycle. If Slave fails to start, Build will fail as well. If slave is slow to start (maybe pulling a huge docker image), build log will report the progress.

etc.

design and use-case : https://wiki.jenkins-ci.org/display/JENKINS/One-Shot+Executor

code : https://github.com/jenkinsci/one-shot-executor-plugin

the current code base do rely on hacks, our goal is to demonstrate this use case (can be tested reusing sample) so we can get the adequate hooks introduced in jenkins-core and later re-implement same API on a cleaner basis.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANMVJznmnydODNX%2BYXmh0ujnJXsoZWcPT%2BpJCWwdR0_wUpfDTQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/WV6sPmIh-tk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/D305926F.28292%25thomas.suckow%40pnnl.gov.

[Jesse Glick]

On Wed, Mar 9, 2016 at 11:52 AM, Suckow, Thomas J
<Thomas...@pnnl.gov> wrote:
> Certain other issues should perform exponential backoff (Cloud
> infrastructure down).

Or just fail the build and the next one should work.

> It could also handle the logic of some users wanting to configure slaves on
> a per job basis.

If you mean “configure Docker images on a per-job basis”, this is
addressed by both the Docker Custom Build Environment and Docker
Pipeline plugins, both of which ought to move toward using this new
infrastructure.

[nicolas de loof]

"

I do agree that certain issues should fail immediately (image not found). Certain other issues should perform exponential backoff (Cloud infrastructure down). Provisioning limits could be annoying though, would be interesting if they could be left in the queue until Jenkins side provisioning limits are not violated. I am not sure how to handle an environment like Kubernetes though where other entities may be utilizing resources and you have to "share".

"

This is something we have in mind. Provisioner could wait for available resources before it creates a Slave, leaving the task in the queue with a LabelAssignment waiting for matching executor. Would anyway need to let the Run start then create the slave, which means some race condition could appear and the required resources aren't available for this Slave to start even they were, few ms before - or we need some way to reserve resources on the infra, which then would significantly limit the available implementations. Maybe then we could cancel the Run, as we run early in it's lifecycle, and re-schedule it as a task in the Queue, claiming the Run never existed ? 

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3ycEVavhfm2Kuj5pOB5fkPF6vaOJfb6Zu-89y%2BQRm95g%40mail.gmail.com.

[nicolas de loof]

please see https://github.com/jenkinsci/one-shot-executor-plugin/commit/86c632091391e3204432a8c7d1a18bb48b3a66e3

basically, this on let the provisionner keep a task in the Queue when it can determine it hasn't enough resources to run extra slaves.

this can be used to prevent overload, or allocate some extra physical nodes.

[Suckow, Thomas J]

On 3/9/16, 12:40 PM, "jenkin...@googlegroups.com on behalf of Jesse
Glick" <jenkin...@googlegroups.com on behalf of jgl...@cloudbees.com>
wrote:

>
>Or just fail the build and the next one should work.

I just don't want to have to kick builds when a normal slave would "just
wait". Our cloud is also far from 99.9% uptime.

[Suckow, Thomas J]

It would be an interesting feature if a build could be returned to the queue. I would actually like that in the case the slave disappeared during the build, I've had that happen a couple times over the last few years. Could also be interesting if you are shutting down jenkins and don't wan't to wait for a build to complete you could abort it and return it to the queue.

I think in the race condition case, as long as only a handful make it into the run state you won't be pounding the server saying "give me a slice." We have some large (to us) matrix builds and it probably isn't good to have 20-30 jobs all trying to get a cut of the pie.

From: <jenkin...@googlegroups.com> on behalf of nicolas de loof <nicolas...@gmail.com>
Reply-To: "jenkin...@googlegroups.com" <jenkin...@googlegroups.com>
Date: Thursday, March 10, 2016 at 1:46 AM
To: "jenkin...@googlegroups.com" <jenkin...@googlegroups.com>

Subject: Re: One-Shot Executors

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANMVJz%3DQNuogSJvuUMWR9RbjczW1fAd8pAJD9aa0Lk0H5aFwAQ%40mail.gmail.com.

[James Nord]

Could also be interesting if you are shutting down Jenkins and don't wan't to wait for a build to complete

 

use pipeline and then your builds will then mostly survive Jenkins restarts.

Or there is a proprietary restart aborted builds plugin that will restart builds that where aborted on Jenkins startup.

[nicolas de loof]

we want to distinguish error due to job misconfiguration (bad docker image ID for sample) and infrastructure issue (no available resources)

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/D30701C3.2858A%25thomas.suckow%40pnnl.gov.