Frustration with build-step/post-build-action access...
317 views
Skip to first unread message
Jeff
Jul 9, 2014, 6:54:26 PM7/9/14
to jenkins...@googlegroups.com
This is a bit of a rant, but I'm unclear why it matters which steps are available as a build step vs. a post-build step.
I'm trying to create a simple job to grab debian packages archived from various upstream "build" jobs and installing/configuring the debian packages on a set of test servers as part of an "install" job.
The "Install job" sequence would be:
- Copy artifacts from previous jobs
- SCP them to the specified server(s)
- Run the installers via SSH
- Update configuration
However, I am unable to use the Jenkins plugins to do step 2 and 3 in the proper order because the "Publish artifacts to SCP Repository" plugin option is not available as a "build step" and "Execute shell script on remote host using ssh" plugin as it's not available as a "post-build action". (chicken or egg paradox).
The motivation to use these plugins is to help manage the hosts/passwords required to use SCP/SSH.
Thus, the question is ... why does Jenkins need to differentiate between a build step and a post-build action? Why not allow a mix of any type of operation/step, regardless if it's type?
It seems job creation flexibility would be enhanced if all tasks could be used, particularly for jobs that don't really do builds (in the traditional sense) such as creating a "pipeline" of phases/steps (continuous delivery, etc.) deploy-dev --> functional test --> deploy-staging --> integration test --> deliver release candidate.
Has there been consideration around allowing any step to be added in any order?
--
Daniel Beck
Jul 9, 2014, 7:22:13 PM7/9/14
to jenkins...@googlegroups.com
Your problem is that you're misusing the 'software build' concept and job type for something that isn't strictly a software build, and its limitations become obvious.
Post-build steps usually fall into one of the categories 'recorder' (e.g. archive artifacts) and 'notifier' (email notifications), things that usually don't make sense before the actual work is done. I expect that very few actual build scripts need to SCP right in the middle, at least compared to those that upload things somewhere when they're finished. So that's what the plugin author decided should be the supported use case, and the reason you have difficulty achieving what you want.
When I try do something like this, I usually try to get all the basic functionality going from a shell/batch script so it's usable outside Jenkins if there's a need to, reducing Jenkins to a fancy scheduler, log archiver and notifier. Passwords are provided by Jenkins as environment variables, SSH keys are stored on the slaves. Versioning of configuration is easier that way, and the job configurations don't become too complex.
That said, have you tried Any Build Step Plugin? It might allow you to do what you want to achieve.
The upcoming Workflow plugin probably also has the required flexibility of combining steps more flexibly in its job type.
> --
> You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Post-build steps usually fall into one of the categories 'recorder' (e.g. archive artifacts) and 'notifier' (email notifications), things that usually don't make sense before the actual work is done. I expect that very few actual build scripts need to SCP right in the middle, at least compared to those that upload things somewhere when they're finished. So that's what the plugin author decided should be the supported use case, and the reason you have difficulty achieving what you want.
When I try do something like this, I usually try to get all the basic functionality going from a shell/batch script so it's usable outside Jenkins if there's a need to, reducing Jenkins to a fancy scheduler, log archiver and notifier. Passwords are provided by Jenkins as environment variables, SSH keys are stored on the slaves. Versioning of configuration is easier that way, and the job configurations don't become too complex.
That said, have you tried Any Build Step Plugin? It might allow you to do what you want to achieve.
The upcoming Workflow plugin probably also has the required flexibility of combining steps more flexibly in its job type.
> You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Jeff
Jul 10, 2014, 12:39:08 AM7/10/14
to jenkins...@googlegroups.com
I wouldn't call it "misusing". I'm simply working with what is there (that I knew about). Since I can't do SSH in post-build I am limited to what I can do, hence my frustration. With the emergence of Continuous Delivery/Deployment, I wish a job could be defined more broadly and tasks or steps within a job more fluidly to include any step in any order without having to resort to writing/installing all sorts of plugins or having to fence an operation to an explicit "type" of job or build phase.
As for the current job definition I've been working with, I would likely do more things in a post-build action if I were allowed to, thus the point of my frustration. I missed the "Any Build Step Plugin" but I will definitely give it a try.
As for using environment variables to handle user/passwords, does that allow restricted access, including obfuscation in build logs? I'm unaware of the mechanism for managing/obfuscating users/passwords in a secure way outside of using the plugins. I'll look into that as well.
Thanks for the feedback and I'm looking forward to trying the Workflow plugin. Is there any information or designs I can look at?
On Wed, Jul 9, 2014 at 5:21 PM, Daniel Beck <m...@beckweb.net> wrote:
Your problem is that you're misusing the 'software build' concept and job type for something that isn't strictly a software build, and its limitations become obvious.
Post-build steps usually fall into one of the categories 'recorder' (e.g. archive artifacts) and 'notifier' (email notifications), things that usually don't make sense before the actual work is done. I expect that very few actual build scripts need to SCP right in the middle, at least compared to those that upload things somewhere when they're finished. So that's what the plugin author decided should be the supported use case, and the reason you have difficulty achieving what you want.
When I try do something like this, I usually try to get all the basic functionality going from a shell/batch script so it's usable outside Jenkins if there's a need to, reducing Jenkins to a fancy scheduler, log archiver and notifier. Passwords are provided by Jenkins as environment variables, SSH keys are stored on the slaves. Versioning of configuration is easier that way, and the job configurations don't become too complex.
That said, have you tried Any Build Step Plugin? It might allow you to do what you want to achieve.
The upcoming Workflow plugin probably also has the required flexibility of combining steps more flexibly in its job type.
On 10.07.2014, at 00:53, Jeff <preda...@gmail.com> wrote:
> This is a bit of a rant, but I'm unclear why it matters which steps are available as a build step vs. a post-build step.
>
> I'm trying to create a simple job to grab debian packages archived from various upstream "build" jobs and installing/configuring the debian packages on a set of test servers as part of an "install" job.
>
> The "Install job" sequence would be:
> * Copy artifacts from previous jobs
> * SCP them to the specified server(s)
> * Run the installers via SSH
> * Update configuration
> --> However, I am unable to use the Jenkins plugins to do step 2 and 3 in the proper order because the "Publish artifacts to SCP Repository" plugin option is not available as a "build step" and "Execute shell script on remote host using ssh" plugin as it's not available as a "post-build action". (chicken or egg paradox).
>
> The motivation to use these plugins is to help manage the hosts/passwords required to use SCP/SSH.
>
> Thus, the question is ... why does Jenkins need to differentiate between a build step and a post-build action? Why not allow a mix of any type of operation/step, regardless if it's type?
>
> It seems job creation flexibility would be enhanced if all tasks could be used, particularly for jobs that don't really do builds (in the traditional sense) such as creating a "pipeline" of phases/steps (continuous delivery, etc.) deploy-dev --> functional test --> deploy-staging --> integration test --> deliver release candidate.
>
> Has there been consideration around allowing any step to be added in any order?
>
> --
> Jeff Vincent
> See my LinkedIn profile at:
> http://www.linkedin.com/in/rjeffreyvincent
>
> You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Baskar Veeramani (baveeram)
Jul 10, 2014, 12:51:05 AM7/10/14
to jenkins...@googlegroups.com
Jeff, we are managing similar requirement by configuring the post build steps as an separate downstream job which triggers automatically when the actual build completes. Copy artifact plugin (Copy artifacts from another project) is used to copy the artifact from build job to downstream job.
Regards,
Baskar.V
Jeff
Jul 10, 2014, 1:29:29 AM7/10/14
to jenkins...@googlegroups.com
Thanks...that's what I am trying to do. I have successfully copied the artifacts. Now I need to SCP them to the server BEFORE I run a remote SSH script to install them.
Using the SCP plugin would help me not have to manually manage credentials in the job or risk exposing them in the logs (am going to investigate other ways per Daniel's comment) but the SCP plugin step is currently only available in a post-build action so I am unable to use it to copy the files before attempting to install them.
I could (and may still) write a local SSH script that simply calls SCP but I'm unsure how to manage credentials securely and still use them in scripts (again, still investigating).
Stephen Connolly
Jul 10, 2014, 3:26:13 AM7/10/14
to jenkins...@googlegroups.com
To help understand, here is what the key difference between build steps and publishers is.
--
Sent from my phone
Publishers are designed for tasks that should run at most once
Archiving artifacts does not make sense to run repeatedly
Emailing build results does not make sense to run repeatedly
SCPing *archived artifacts* is thus a special case because it relies on the existence of archived artifacts.
If I were designing the SCP plugin I would have a special publisher to publish *archived artifacts* AND a build step to SCP files *from the workspace*
The current plugin is missing that second class of action.
Wrt your current issue... Perhaps the ssh-agent plugin would solve your problem as then a simple shell script could just call SCP
--
Sent from my phone
Daniel Beck
Jul 10, 2014, 6:33:27 AM7/10/14
to jenkins...@googlegroups.com
> I wouldn't call it "misusing". I'm simply working with what is there (that I knew about).
… for a problem 'what is there' was not designed to solve. Law of the instrument ;-)
> Since I can't do SSH in post-build I am limited to what I can do, hence my frustration. With the emergence of Continuous Delivery/Deployment, I wish a job could be defined more broadly and tasks or steps within a job more fluidly to include any step in any order without having to resort to writing/installing all sorts of plugins or having to fence an operation to an explicit "type" of job or build phase.
> Thanks for the feedback and I'm looking forward to trying the Workflow plugin. Is there any information or designs I can look at?
https://github.com/jenkinsci/workflow-plugin
Slides of "Workflow in Jenkins" at https://www.cloudbees.com/jenkins/juc-2014/boston/sessions
Also, the jenkinsci-dev list archives should have various bits about it.
> As for the current job definition I've been working with, I would likely do more things in a post-build action if I were allowed to, thus the point of my frustration. I missed the "Any Build Step Plugin" but I will definitely give it a try.
>
> As for using environment variables to handle user/passwords, does that allow restricted access, including obfuscation in build logs? I'm unaware of the mechanism for managing/obfuscating users/passwords in a secure way outside of using the plugins. I'll look into that as well.
Les Mikesell
Jul 10, 2014, 1:15:17 PM7/10/14
to jenkinsci-users
On Thu, Jul 10, 2014 at 12:29 AM, Jeff <preda...@gmail.com> wrote:
> Thanks...that's what I am trying to do. I have successfully copied the
> artifacts. Now I need to SCP them to the server BEFORE I run a remote SSH
> script to install them.
>
> Using the SCP plugin would help me not have to manually manage credentials
> in the job or risk exposing them in the logs (am going to investigate other
> ways per Daniel's comment) but the SCP plugin step is currently only
> available in a post-build action so I am unable to use it to copy the files
> before attempting to install them.
>
> I could (and may still) write a local SSH script that simply calls SCP but
> I'm unsure how to manage credentials securely and still use them in scripts
> (again, still investigating).
Not sure I understand the point of copying things twice if the jenkins
> Thanks...that's what I am trying to do. I have successfully copied the
> artifacts. Now I need to SCP them to the server BEFORE I run a remote SSH
> script to install them.
>
> Using the SCP plugin would help me not have to manually manage credentials
> in the job or risk exposing them in the logs (am going to investigate other
> ways per Daniel's comment) but the SCP plugin step is currently only
> available in a post-build action so I am unable to use it to copy the files
> before attempting to install them.
>
> I could (and may still) write a local SSH script that simply calls SCP but
> I'm unsure how to manage credentials securely and still use them in scripts
> (again, still investigating).
node doesn't actually do anything to the files. Can't your target
server script just grab the archived artifacts directly from the
jenkins web interface itself? In any case, if the point is just to
run some arbitrary stuff via ssh from a node with strategic firewall
access, why not use the ssh-agent plugin and do whatever you need in
one or more build scripts that can be embedded in the job or pulled
from an scm? A side benefit is that you can use rsync over ssh
instead of scp which can sometimes be helpful.
--
Les Mikesell
lesmi...@gmail.com
Reply all
Reply to author
Forward
0 new messages