SGE -b plug-in argument

[Fred Loney]

The Nipype SGE plug-in documentation example does not include '-b n' in the workflow run call. Our local cluster admin suggests always including '-b n' in the workflow run plugin_args. -b n indicates to SGE that the execution script is not binary. SGE then parses the script for submission directive lines which begin with '#$'. I notice that the default Nipype SGE template consists of the following:

#$ -V

#$ -S /bin/sh

'-V' pulls in all environment variables, and is perhaps significant. This suggests that SGE-enabled workflows should always be run as follows:

workflow.run(plugin='SGE', plugin_args={'qsub_args' : '-b n'})

When I use this call, Nipype succeeds on script interface commands but fails on binary interface commands. E.g. the fsl interfaces fails with an error message buried deep in the batch directory that 'fslmaths cannot be found'. When I don't include '-b n', the workflow succeeds.

It appears that the workflow run plugin_args serve as the default qsub options for every submitted workflow task. Is that correct?

If I don't include '-b n' in plugin_args, then SGE ignores the submission directives. Is that correct? If so, is there a danger in ignoring the -V directive? I suspect not, unless perhaps the user modifies the login env vars before submission. Even then, I think SGE replicates the qsub environment in each job.

Each Nipype task submits a batch script which copies in the template. This script then calls a python script generated by Nipype. If that is the case, then why would '-b n' ever fail since it is the batch script which is called, not the interface command?

I appreciate in advance any clarification.

Fred

[Satrajit Ghosh]

hi fred,

The Nipype SGE plug-in documentation example does not include '-b n' in the workflow run call. Our local cluster admin suggests always including '-b n' in the workflow run plugin_args. -b n indicates to SGE that the execution script is not binary. SGE then parses the script for submission directive lines which begin with '#$'. I notice that the default Nipype SGE template consists of the following:

#$ -V

#$ -S /bin/sh

'-V' pulls in all environment variables, and is perhaps significant. This suggests that SGE-enabled workflows should always be run as follows:

workflow.run(plugin='SGE', plugin_args={'qsub_args' : '-b n'})

When I use this call, Nipype succeeds on script interface commands but fails on binary interface commands. E.g. the fsl interfaces fails with an error message buried deep in the batch directory that 'fslmaths cannot be found'. When I don't include '-b n', the workflow succeeds.

i've never used the -b n option, but according to the qsub manpage: "qsub will implicitly use -b n"

 

It appears that the workflow run plugin_args serve as the default qsub options for every submitted workflow task. Is that correct?

yes - unless you override it with: 

node.plugin_args = {'overwrite': True, 'qsub_args': '-l host=somehost'}

 

If I don't include '-b n' in plugin_args, then SGE ignores the submission directives. Is that correct?

doesn't seem so from the SGE manpage. 

 

If so, is there a danger in ignoring the -V directive? I suspect not, unless perhaps the user modifies the login env vars before submission. Even then, I think SGE replicates the qsub environment in each job.

the other scenario where the -V directive would be ignored is if the user has a bashrc/profile that strips out the component related to whether the user is on an interactive or no-interactive shell. the default bash profiles on linux systems distinguish between interactive and non-interactive logins.

 

Each Nipype task submits a batch script which copies in the template. This script then calls a python script generated by Nipype.

 

If that is the case, then why would '-b n' ever fail since it is the batch script which is called, not the interface command?

it is the case that batch script calls a python script. however, i don't know the answer to why the command would fail. perhaps it might be worthwhile checking the qsub outputs (.o* and .e* files).

cheers,

satra

[Brendan Moloney]

Hello,

Since I am the cluster admin Fred mentioned, figured I should throw my 2 cents in. 

The default behavior of SGE is indeed to use "-b n".  However this requires users to pass the full path for any commands they want to qsub, even if they are on the PATH (e.g. you would be required to do "qsub /usr/lib/fsl/5.0/bet my_image.nii" instead of just "qsub bet my_image.nii"). Therefore, in our cluster we set the default to "-b y" in the global "sge_request" file.  The downside to this is that users who want to use embedded directives will need to specify "-b n" (we find this an acceptable trade off on our cluster as embedded directives are used pretty rarely, and when they are used it is typically by our more experienced users).

As Fred mentioned Nipype does use embedded directives and thus on our cluster the "-b n" option is needed.  I have never run into problems caused by including this option, but the only possible cause I can think of would be the PATH issues discussed above. Ideally, I think Nipype should either 1) always use the "-b n" option and make sure to provide full paths for commands or 2) not make use of embedded directives and instead just pass the options on the command line.

Thanks,

Brendan

--
 
---
You received this message because you are subscribed to the Google Groups "NiPy Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nipy-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Fred Loney]

Satra and Brendan,

Thanks for the responses. I confirmed that, at least in our environment, the only perceived effect of the -V directive is to include environment variables that are defined after login. -b y (implicit or not) still pulls in the login environment. Since I do not change the environment after login, this has no effect for me. As Brendan notes, -b n will fail if the submitted command is not a real path. However, Nipype only submits the shell script that it builds, which in turn runs the interface, which in turn calls the wrapped command, e.g. fslmaths. Thus, the -b setting should have no effect on execution in my case. Therefore, the observed error remains a mystery. I moved past the error and can't reproduce it in a simpler example, so I will chalk it up to a transitory aberration.

The upshot is that AFAIK the -b flag is irrelevant except in the following situations:

• an environment variable or alias is set after login which changes the path resolution
  
• the user sets the Nipype node plugin_args template to include additional directives

Nipype already validates that a command is on the path in CommandLine._run_interface, but doesn't use the path in the command. (As an aside, this validation will incorrectly fail if the command is defined by an alias).

I suggest the following, as a low priority:

• Nipype remove the -S and -V directives in the batch script, since they have limited and uncertain effect.
• Nipype neither assume that the site's SGE has a vanilla setup nor set the -b n option.
• Add an SGE example to the documentation that defines a plugin_args template with directives. The example should include a note that if the user site has changed the SGE default behavior to -b y, then the plugin_args should include the -b n option.

The Nipype submission should exactly reflect a standard qsub submission in the user's environment. If the user needs to add an SGE option or embed a directive, then the user should set the plugin_args.

I find embedded SGE directives too much magic for my taste, and consider it a best practice to avoid them when possible.

Fred

[Satrajit Ghosh]

thanks for the summary fred.

i'll add one small note in relation to our setup (which used to be SGE a while back). the reason -V is necessary in our setting is that our bash_profile (and this is the default on many unix systems) effectively looks like this:

```

# If not running interactively, we're done

[ -z "$PS1" ] && return

source ~/.bashrc

```

what this means is that nothing in bashrc gets executed when the process logs in to a job node. The -V flag ensures that the local environment is used in the remote job (things like path settings to python, fsl, etc.,.). 

how is your environment different from the above? is it because of how users have customized or the center has customized the setup? 

The upshot is that AFAIK the -b flag is irrelevant except in the following situations:

• an environment variable or alias is set after login which changes the path resolution
  
• the user sets the Nipype node plugin_args template to include additional directives

Nipype already validates that a command is on the path in CommandLine._run_interface, but doesn't use the path in the command.

correct - i want to slowly think about moving to containers for running nipype jobs and it will be interesting to figure out

 

(As an aside, this validation will incorrectly fail if the command is defined by an alias).

let's hope aliases are not used at least for the base interfaces from packages.

 

I suggest the following, as a low priority:

• Nipype remove the -S and -V directives in the batch script, since they have limited and uncertain effect.
• Nipype neither assume that the site's SGE has a vanilla setup nor set the -b n option.
• Add an SGE example to the documentation that defines a plugin_args template with directives. The example should include a note that if the user site has changed the SGE default behavior to -b y, then the plugin_args should include the -b n option.

The Nipype submission should exactly reflect a standard qsub submission in the user's environment. If the user needs to add an SGE option or embed a directive, then the user should set the plugin_args. 

I find embedded SGE directives too much magic for my taste, and consider it a best practice to avoid them when possible.

that's why we have the most limited set. i'd like to keep the -V. in most clusters users would have to specify that. but i agree we need to improve docs and perhaps add something to suppress -V. if we find that -V is not useful to most users, we should remove it, but my current impression is that it is. the -S doesn't affect people because that's specific to the fact that nipype writes out the script to execute.

cheers,

satra

[Fred Loney]

Hi Satra,

I see your point. My experience is that unconditionally sourcing .bashrc from .bash_profile is far more common, e.g. that is what Ubuntu and Gnu recommend. Without getting into that interminable debate, one of the reasons is that opening a console only reads .bashrc, not .bash_profile.

In the SGE environment you describe, you presumably always had to call qsub with the -V option. As a convenience, Nipype relieves the user in this environment of specifying -V in the plugin_args.

There is a transparency vs. convenience trade-off: should Nipype reflect the user's environment or improve on the user's environment? As a rule, execution frameworks should favor transparency. A rule of thumb for the user should be "Whatever you set in a qsub, put that in your qsub_args.' However, in this case, it is no big deal either way. Embedding -V is fairly harmless and saves some users three keystrokes.

Fred