Copy files from remote Jenkins workspace

1,265 views
Skip to first unread message

buddy123

unread,
Nov 13, 2016, 5:23:07 AM11/13/16
to Jenkins Users

Im trying to create a cross-compilation job, that will build my code for both Windows & Linux. I have set up Windows & Linux jenkins nodes, and configured two jobs, one with 'windows' label to run on my Windows machine and the other with 'linux' label to run on my linux machine. I installed MultiJob plugin, and wrapped both jobs to run in parallel When both finishes successfully, I want to copy all the generated files and do operations on them In other words I want to copy the files from both jobs remote workspaces for further operations Can anyone suggest a way how to do it?

Baptiste Mathus

unread,
Nov 13, 2016, 9:40:22 AM11/13/16
to jenkins...@googlegroups.com
Look for pipeline jobs. Might be far easier to achieve that.


2016-11-13 11:23 GMT+01:00 buddy123 <er...@twistlock.com>:

Im trying to create a cross-compilation job, that will build my code for both Windows & Linux. I have set up Windows & Linux jenkins nodes, and configured two jobs, one with 'windows' label to run on my Windows machine and the other with 'linux' label to run on my linux machine. I installed MultiJob plugin, and wrapped both jobs to run in parallel When both finishes successfully, I want to copy all the generated files and do operations on them In other words I want to copy the files from both jobs remote workspaces for further operations Can anyone suggest a way how to do it?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/05639dc0-fce6-4ce7-9cc1-c39e95ca7498%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jovan Jevric

unread,
Nov 14, 2016, 4:12:08 AM11/14/16
to jenkins...@googlegroups.com
What about archiving artefacts. On that way all output marked as artefacts could be found on master so far, couldn't it?
 
 
Original Message processed by david®  
 
Re: Copy files from remote Jenkins workspace (13-Nov-2016 15:39)
From:   Baptiste Mathus
To:
 jenkins...@googlegroups.com
 
Look for pipeline jobs. Might be far easier to achieve that.

 
2016-11-13 11:23 GMT+01:00 buddy123 <er...@twistlock.com>:

Im trying to create a cross-compilation job, that will build my code for both Windows & Linux. I have set up Windows & Linux jenkins nodes, and configured two jobs, one with 'windows' label to run on my Windows machine and the other with 'linux' label to run on my linux machine. I installed MultiJob plugin, and wrapped both jobs to run in parallel When both finishes successfully, I want to copy all the generated files and do operations on them In other words I want to copy the files from both jobs remote workspaces for further operations Can anyone suggest a way how to do it?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

buddy123

unread,
Nov 14, 2016, 4:15:54 AM11/14/16
to Jenkins Users
I'll look into pipeline, though it seems very complicated.
Regarding artifacts, I've thought about it, but I cant figure out how to delete the artifacts once I finished the job, otherwise my disk space will get empty very quickly.
Is it possilble to mark files as artifacts, pull them from slave, and then erase them?
I want to keep the console output of every build though, just dont care about the artifacts because I store them elsewhere


On Monday, November 14, 2016 at 11:12:08 AM UTC+2, jje wrote:
What about archiving artefacts. On that way all output marked as artefacts could be found on master so far, couldn't it?
 
 
Original Message processed by david®  
 
Re: Copy files from remote Jenkins workspace (13-Nov-2016 15:39)
From:   Baptiste Mathus
To:
 jenkins...@googlegroups.com
 
Look for pipeline jobs. Might be far easier to achieve that.

 
2016-11-13 11:23 GMT+01:00 buddy123 <er...@twistlock.com>:

Im trying to create a cross-compilation job, that will build my code for both Windows & Linux. I have set up Windows & Linux jenkins nodes, and configured two jobs, one with 'windows' label to run on my Windows machine and the other with 'linux' label to run on my linux machine. I installed MultiJob plugin, and wrapped both jobs to run in parallel When both finishes successfully, I want to copy all the generated files and do operations on them In other words I want to copy the files from both jobs remote workspaces for further operations Can anyone suggest a way how to do it?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/05639dc0-fce6-4ce7-9cc1-c39e95ca7498%40googlegroups.com[groups.google.com].
For more options, visit https://groups.google.com/d/optout[groups.google.com].

Jovan Jevric

unread,
Nov 14, 2016, 4:40:44 AM11/14/16
to jenkins...@googlegroups.com
Do archive the artefacts elsewhere by running a post-build job, as you proposed. And simply parametrize your job that way, that elder jobs on your host are being deleted automatically. In other words - simply discard old builds, doing that is very simple.
 
 
 
Original Message processed by david®  
 
Re: Re-2: Copy files from remote Jenkins workspace (14-Nov-2016 10:15)
From:   buddy123
To:
 Jenkins Users

Jovan Jevric

unread,
Nov 14, 2016, 4:41:08 AM11/14/16
to jenkins...@googlegroups.com
...post build job or post build step and executing a shell script or similar, which moves your artefacts.
 
 
Original Message processed by david®  
 
Re: Re-2: Copy files from remote Jenkins workspace (14-Nov-2016 10:15)
From:   buddy123
To:
 Jenkins Users
 

Eran Yanay

unread,
Nov 14, 2016, 4:42:07 AM11/14/16
to jenkins...@googlegroups.com
Discarding older builds means it discards my console output of these builds as well, I wish to keep it.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/1OChk_e4A6Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/DIIE.0000228E0093FFA4%4010.11.0.23.
For more options, visit https://groups.google.com/d/optout.

Jovan Jevric

unread,
Nov 14, 2016, 5:14:40 AM11/14/16
to jenkins...@googlegroups.com
You'll have to do some housekeeping anyway....Otherwise you'll run out of memory very fast.
By archiving all the artefacts generated(mainly needed for release steps or similar) on your own you'll be able to keep your jenkins workspace very slim. This is an advantage and practicable.
 
Keeping Console Output by simply archiving is possible. I do so on the following way:
-Use Log Parser Plugin which generates a Log.txt or similar in your project on slave workspace and include it as post build step('Console output(build log)parsing')
-Get your Console Output from slaves by scp/ssh or similar e.g.
 
 
Original Message processed by david®  
 
Re: Re-4: Copy files from remote Jenkins workspace (14-Nov-2016 10:41)
From:   Eran Yanay
To:
 jenkins...@googlegroups.com

To unsubscribe from this group and all its topics, send an email to jenkinsci-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

Eran Yanay

unread,
Nov 14, 2016, 6:42:03 AM11/14/16
to jenkins...@googlegroups.com
My workspace is already very slim, because I override it for every build.
So I dont really have any housekeeping, as each build only stores information like console output, and since all my work is done inside the workspace, I never go out of space.
If I now make artifacts of my workspace for every build, Is it possible to remove only the artifact once its downloaded to my master node?

On Mon, Nov 14, 2016 at 12:14 PM, Jovan Jevric <j...@huber-online.com> wrote:
You'll have to do some housekeeping anyway....Otherwise you'll run out of memory very fast.
By archiving all the artefacts generated(mainly needed for release steps or similar) on your own you'll be able to keep your jenkins workspace very slim. This is an advantage and practicable.
 
Keeping Console Output by simply archiving is possible. I do so on the following way:
-Use Log Parser Plugin which generates a Log.txt or similar in your project on slave workspace and include it as post build step('Console output(build log)parsing')
-Get your Console Output from slaves by scp/ssh or similar e.g.
 
 
Original Message processed by david®  
 
Re: Re-4: Copy files from remote Jenkins workspace (14-Nov-2016 10:41)
From:   Eran Yanay

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/1OChk_e4A6Y/unsubscribe.

To unsubscribe from this group and all its topics, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/DIIE.000021EC00940142%4010.11.0.23.
For more options, visit https://groups.google.com/d/optout.

Jovan Jevric

unread,
Nov 15, 2016, 1:22:01 AM11/15/16
to jenkins...@googlegroups.com
Workspace per definition is space you gave jenkins to store all the builds executed on master. If you run 50 Jobs from JobA all builds including artefacts will be stored on master. So you'll run out of memory defenitelly if you don't care about housekeepings. Local workspace you mean is workspace acquired on jenkins slave for a job execution. Keep on working on your basics dude;-)
 
 
Don't archive artefacts and get the artefacts from slave on your own. On that way none artefacts are put on your master machine. But keep in mind, that none artefacts can be accessed in your job history if not archived.
 
Cheers
 
 
Original Message processed by david®  
 
Re: Re-6: Copy files from remote Jenkins workspace (14-Nov-2016 12:41)
From:   Eran Yanay
To:
 jenkins...@googlegroups.com

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

Baptiste Mathus

unread,
Nov 16, 2016, 2:56:21 PM11/16/16
to jenkins...@googlegroups.com
Hello Jovan,

Thank you very much for your help in general. It's much appreciated. 
Could you please see to configure your mailer so that it does not change the subject by prefixing it by "Re'-" and the email number or something? I does create a new thread locally in Gmail for me, so I guess I'm not the same (though the WebUI of the googlegroups seems to handle that).

Thank you 

2016-11-15 7:21 GMT+01:00 Jovan Jevric <j...@huber-online.com>:
Workspace per definition is space you gave jenkins to store all the builds executed on master. If you run 50 Jobs from JobA all builds including artefacts will be stored on master. So you'll run out of memory defenitelly if you don't care about housekeepings. Local workspace you mean is workspace acquired on jenkins slave for a job execution. Keep on working on your basics dude;-)
 
 
Don't archive artefacts and get the artefacts from slave on your own. On that way none artefacts are put on your master machine. But keep in mind, that none artefacts can be accessed in your job history if not archived.
 
Cheers
 
 
Original Message processed by david®  
 
Re: Re-6: Copy files from remote Jenkins workspace (14-Nov-2016 12:41)
From:   Eran Yanay

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/DIIE.000101E2009413D8%4010.11.0.23.
For more options, visit https://groups.google.com/d/optout.

Reply all
Reply to author
Forward
0 new messages