Setting a group of params with mode-specific default values

[Owen S.]

I have a large number of params that need to be set for different "modes" of my pipeline.

// mode 1:

params.a = 1

params.b = 2

params.c = 3

(etc)

// mode 2:

params.a = 10

params.b = 20

params.c = 30

(etc)

I would like to put default values for all the params into the nextflow.config file, and then somehow activate the values according to the mode that is requested.  This way, instead of specifying all the parameters in a job-specific config file, I can just specify the "mode".  I can't figure out a clean way of doing this.  

I thought of putting the mode specific params in a separate config file, and then from my job-specific config file, I had something like:  

includeConfig '${workflow.projectDir}/mode1.config' 

but apparently workflow introspection isn't available when config files are being evaluated.  I'd prefer not to resort to absolute paths if possible.

Perhaps there is another approach I am not thinking about.  Any suggestions would be much appreciated! Thanks.

[Paolo Di Tommaso]

Have you tried config profiles? 

https://www.nextflow.io/docs/latest/config.html#config-profiles

p

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/nextflow.
For more options, visit https://groups.google.com/d/optout.

[Owen S.]

Thank you for the suggestion. I thought of this, but I'm already using profiles to specify executor level parameters, much like the example you provide in the online docs.    So unless it's possible to specify multiple profiles in the same run, I don't think this is a clean solution for me.  

On Tuesday, June 20, 2017 at 2:50:10 PM UTC-7, Paolo Di Tommaso wrote:

Have you tried config profiles? 

https://www.nextflow.io/docs/latest/config.html#config-profiles

p

On Tue, Jun 20, 2017 at 9:19 PM, Owen S. <owen.s...@gmail.com> wrote:

I have a large number of params that need to be set for different "modes" of my pipeline.

// mode 1:

params.a = 1

params.b = 2

params.c = 3

(etc)

// mode 2:

params.a = 10

params.b = 20

params.c = 30

(etc)

I would like to put default values for all the params into the nextflow.config file, and then somehow activate the values according to the mode that is requested.  This way, instead of specifying all the parameters in a job-specific config file, I can just specify the "mode".  I can't figure out a clean way of doing this.  

I thought of putting the mode specific params in a separate config file, and then from my job-specific config file, I had something like:  

includeConfig '${workflow.projectDir}/mode1.config' 

but apparently workflow introspection isn't available when config files are being evaluated.  I'd prefer not to resort to absolute paths if possible.

Perhaps there is another approach I am not thinking about.  Any suggestions would be much appreciated! Thanks.

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.

[Paolo Di Tommaso]

Multiple profile, you could create a profile for each combination of executor and *mode*. Using the includeConfig it not so verbose: 

param.foo = 'def_value'

params.bar = 'other_def' 

profiles {

  mode1_local {

      process.executor = 'local'

      includeConfig 'mode1.config'

  }

  mode1_sge {

       process.executor = 'sge'

       includeConfig 'mode1.config' 

  }

  .. etc

}

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.

[Owen S.]

Thanks -- I'd thought of this solution, but I have a lot of "modes" so would prefer not to overload my profiles.

As an alternative, is it possible to read in a config file from within the pipeline.nf file?  In other words, can the .nf file open and apply the settings in an arbitrary config file?  That way, I could just read the appropriate config file, conditional on some if-else statements.

Thanks

[Paolo Di Tommaso]

Well, you could something line this in your script eventually with a conditional statement" 

def cfg = file('mode1.config').text

def mode1 = new ConfigSlurper().parse(cfg)

params.putAll(mode1)

However any params in the nextflow config file have an higher priority. 

p

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.

[Owen S.]

Thank you, this a nice solution.  And thanks for the heads up about config priority order.

Best

Owen

[Steve]

You can now stack multiple profiles in your workflow, so its much easier to have separate profiles with configs for executors and configs for mode and then just call the appropriate combination when you run the pipeline

[Owen S.]

Steve -- thanks so much for pointing this out.   (For anyone stumbling across this thread, this feature was added in release 0.28.0)

And of course, thank you Paolo for the improvements!