Pipeline : Confused about file access on slaves

[Jonathan Hodgson]

Hi,

I've touched on this stuff in a previous thread where I thought it was about running on Windows, in turns out it's the same for any slave

So I'm starting a new thread now I'm clearer what the issue is

I have code that basically works like this

node('master') {  // Master is running Linus

    SET_STUFF_UP

}           

platform_builders ["Mac"] = {

    node ("XCode")

    {

        BUILD_STUFF_ON_OSX

    }

}

platform_builders ["Windows"] = { 

    node ("VS2013")

    {

        BUILD_STUFF_ON_WINDOWS

    }

}

parallel(platform_builders)

So, pretty standard.

Now what I don't understand is what I get when I try to access files.

In SET_STUFF_UP jenkins Buildsteps that access files (checkouts, DeleteDir, etc) operate on the master (as expected) and Groovy's standard file handling classes (File, FileNameFinder etc) also act on the master... so that's all fine

but in BUILD_STUFF_IN_OSX and BUILD_STUFF_IN_WINDOWS buildsteps operate on files on the slaves (as expected) while Groovy standard classes operate on the master.

1) Is this expected? The documentation says that File operates on "the current machine", which I took to mean the one actually running the code at that moment, which is the slave. But that isn't hwat is happening

2) How do I get around it? Buildsteps just don't provide the functionality I need, and doing stuff in shell scripts and batch files (which works as expected) is really clunky.

 

[Jonathan Hodgson]

Further investigation..

I thiought the solution might lie in FilePath

as per here

http://stackoverflow.com/questions/30608444/in-jenkins-job-create-file-using-system-groovy-in-current-workspace/32514581#32514581

But it seems that doesn't apply to Groovy in the pipeline, If I try to declare a FilePath in a pipeline script I get "unable to resolve class FilePath"

I'm surprised that all my searching has failed to being up any information on this issue.

Surely I can't be the only person actually needs to manipulate files on the slave in a pipeline script beyond the very simple options provided in pipeline build steps?

[Daniel Beck]

> On 18.10.2016, at 00:58, Jonathan Hodgson <j.r.h...@gmail.com> wrote:
>
> The documentation says that File operates on "the current machine", which I took to mean the one actually running the code at that moment, which is the slave.

The script _is_ evaluated on the master.

FilePath would do it, or just not using Pipeline as a general purpose scripting language (it's not meant to be). Write a batch/shell script instead if you need to prepare the workspace.

[Jonathan Hodgson]

Hi Daniel,

thanks for trying to help, however....

On Tuesday, October 18, 2016 at 12:28:18 PM UTC+1, Daniel Beck wrote:

> On 18.10.2016, at 00:58, Jonathan Hodgson <j.r.h...@gmail.com> wrote:
>
> The documentation says that File operates on "the current machine", which I took to mean the one actually running the code at that moment, which is the slave.

The script _is_ evaluated on the master.

I'd worked out that this must be the case, but it wasn't clear at first. It doesn't help that when I search for information on these things I quite often come up with stuff that was written with reference to a system.groovy builld step, rather than pipeline, and don't always realize when this is the case.

 

FilePath would do it,

Did you miss my last post? I tried FilePath, but I get a "unable to resolve class" error. 

And if that worked, I still wouldn't know where to get the "channel" parameter from, since build.workspace.channel  is not valid in pipelines.

Is there any example code for this? It may be simple when you know how, but when you don't, it isn't. And finding an answer in the sparse and disjointed documentation isn't easy.

 

or just not using Pipeline as a general purpose scripting language (it's not meant to be).

Why not? It seems like the obvious place to do auto-build specific work. It seems to me that it should be an aim of the project to allow it.

Write a batch/shell script instead if you need to prepare the workspace.

Frankly, I find that a bit of a ridiculous situation. I'm in a powerfull object oriented cross platform language, but to do anything useful I have to drop into platform specific archaic shell environments? Writing clunky code which has to be different for each platform (especially since I've had zero success getting sh to work on windows, despite having cygwin, there's another thread about that on here somewhere).

[Jonathan Hodgson]

Ok, well it seems the "unable to resolve class" issue was due to me not importing the class.

Got thrown by the fact that File is there, hadn't really clicked that FilePath was a hudson specific class rather than standard Groovy.

Still looking for how to find the channel

[Jesse Glick]

On Tuesday, October 18, 2016 at 8:09:56 AM UTC-4, Jonathan Hodgson wrote:

Did you miss my last post? I tried FilePath, but I get a "unable to resolve class" error.

Do not attempt to use `FilePath` from Pipeline script. It will not work.

I'm in a powerfull object oriented cross platform language, but to do anything useful I have to drop into platform specific archaic shell environments? Writing clunky code which has to be different for each platform

There are plenty of platform-independent scripting languages out there. Ruby, Python, etc. Groovy (run as an external process!), for that matter.

I've had zero success getting sh to work on windows

Use `bat` on Windows. There is a `isUnix` step that may be used to select a Unix-vs.-Windows variant.

[Jonathan Hodgson]

On Tuesday, October 18, 2016 at 3:21:09 PM UTC+1, Jesse Glick wrote:

On Tuesday, October 18, 2016 at 8:09:56 AM UTC-4, Jonathan Hodgson wrote:

Did you miss my last post? I tried FilePath, but I get a "unable to resolve class" error.

Do not attempt to use `FilePath` from Pipeline script. It will not work.

Thanks, You've saved me a lot of wasted time trying to find how to make it work, though I've already lost way too much. I really wish this stuff was documented. 

it also begs the question "Why?", Iask because I was looking at pipeline step plugins to see if I could find clues as to how to get the channel, and I see that FilePath is how they work. So it seems that FilePaths are working in the pipeline, just not in Groovy in the pipeline.

 

I'm in a powerfull object oriented cross platform language, but to do anything useful I have to drop into platform specific archaic shell environments? Writing clunky code which has to be different for each platform

There are plenty of platform-independent scripting languages out there. Ruby, Python, etc. Groovy (run as an external process!), for that matter.

Yes, but even then things are messy. All I'm doing is checking for a certain set of subfolders, comparing them with a list, and deleting the ones which aren't on the list.

Now on the master, where File works, that's actually quite a simple task thanks to the power of Groovy. A few lines of code right there where they're needed. 

But it seems to do the same thing on the slave, I basically have to write a separate application and call it through sh or bat, passing parameters to it (either as environment variables or as command line arguments I guess, neither of which is very clean).

One way is simple, neat and easy to maintain. The other is ugly and clunky.

Can you understand why I think that's a significant omission in the design of the pipeline?

When you say "groovy run as an external process", do you mean calling it through bat/sh, or is there a cleaner way?

 

I've had zero success getting sh to work on windows

 

Use `bat` on Windows. There is a `isUnix` step that may be used to select a Unix-vs.-Windows variant.

That's what I've been doing. But since I repeatedly have to execute effectively the same thing on windows and Unix, that's a lot of isUnix, and almost repeated code (almost because shell and bat are not compatible, obviously)

Now it looks like sh is supposed to be supported on windows if I have a shell installed, so I tried using that, but I couldn't get it to work. It called the shell ok (though even there   there's an issue as I recall, since the shell path configuration setting is a global one, but my unix machines don't have shell in the same place as the windows ones) but it didn't seem to be able to actually find the shell script that was generated.

All these things can be worked around of course, but every work around is another bit of bloat which makes things harder to understand and maintain.

Thanks again for letting me know about FilePath being a dead end

[Baptiste Mathus]

One thing to understand is that your pipeline is actually always to be run from/on the master. Each "instruction" if actually sent from the master the requested node, then kind of checkpointed before going further.

So, when you're using a pipeline, it's aware of that. When you're using directly groovy, you end up running on the master.

That's one the reasons you should prefer using the pipeline-utility-steps plugin for that matter, and in general keep your pipeline code as simple as possible.

HTH

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/f314bd0d-7fea-469e-8616-8ad60ce1885f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jonathan Hodgson]

On Friday, October 21, 2016 at 6:11:28 PM UTC+1, Baptiste Mathus wrote:

One thing to understand is that your pipeline is actually always to be run from/on the master. Each "instruction" if actually sent from the master the requested node, then kind of checkpointed before going further.

So, when you're using a pipeline, it's aware of that. When you're using directly groovy, you end up running on the master.

That's one the reasons you should prefer using the pipeline-utility-steps plugin for that matter, and in general keep your pipeline code as simple as possible.

HTH

Hi Baptiste,

thanks, I'd pretty much worked that out but it's good to have it clarified.

Having gone through various hassles relating to Groovy in the pipeline I think I might propose a "Gotchas" section for the documentation, pointing out the pitfalls. It's confusing when you're starting out and some things work while others don't, and then you google for answers to questions and don't realize that the answers you've found relate to system.groovy rather than pipeline.groovy. A simple "this is the issue, this works, this doesn't (or cannot be guaranteed to), beware the difference with system.groovy when looking for answers to questions" page would have saved me an incredible amount of time.

Also, I wonder if there would be a way to implement a pipeline step that let you run groovy on the slave, passing parameters to it. Doing stuff through shell scripts, even if you're basically using them to call scripts in other languages, is rather messy.