Simple question on Dynamic resource allocation in config file

[Sayantan Das]

Hi, I am not sure if I am doing something silly, but it seems that we can dynamically assign the attribute `errorStrategy` but we cannot do the same for `clusterOptions`. The following is throwing an error. Any help would be appreciate. It runs fine if I highlight the clusterOptions

process {

    withLabel: sge_label {

        errorStrategy       = { task.exitStatus in 137..140 ? 'retry' : 'terminate' }

        clusterOptions      = "-l h_vmem="+task.exitStatus.toString().replaceAll(/[\sB]/,'')

        maxRetries          = 3

    }

}

ERROR: 

N E X T F L O W  ~  version 20.01.0

Launching `/home/sayantand/sci/analyses/hla_imputation_v2/src/nf/main.nf` [insane_sanger] - revision: 63e27b4ec0

Unknown config attribute `process.withLabel:sge_label.task.exitStatus` -- check config file: /home/sayantand/sci/analyses/hla_imputation_v2/src/nf/nextflow.config

[Alan Hoyle]

Is there a reason you’re using clusterOptions to specify h_vmem rather than using process.memory and letting Nextflow figure out which options to send the executor?

--

Alan Hoyle - al...@alanhoyle.com - alanhoyle.com

---

From: 'Sayantan Das' via Nextflow <next...@googlegroups.com>
Sent: Tuesday, June 9, 2020 3:43:15 AM
To: Nextflow <next...@googlegroups.com>
Subject: Simple question on Dynamic resource allocation in config file

 

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/9238a4b9-f996-43bb-8501-a54c9714e5ffo%40googlegroups.com.

[Sayantan Das]

Yes, for SGE, the process memory doesn't actually assign v_mem but it assigns free_memory (or some other SGE variable). I think the authors suggested using cluster options it use SGE memory parameter. 

You received this message because you are subscribed to a topic in the Google Groups "Nextflow" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nextflow/mS5Lw4meg6c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nextflow+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/DM6PR12MB383555B5FBF9934A3533271CA0820%40DM6PR12MB3835.namprd12.prod.outlook.com.

[Alan Hoyle]

As an aside, I know you can manipulate the memory request based on the task.attempt.  Here is an example derived from something we do:

process task_that_occasionally_needs_more_ram {

    cpus 8

    memory {16.GB * task.attempt }

    errorStrategy {task.attempt < 4 ? 'retry' : 'finish }

    maxRetries 4

    

    input: [...]
}

We let nextflow handle the details on how the memory allocations are specified for the executor.  We use SLURM primarily, but we used to use SGE before we moved to Nextflow.  

Also, wouldn't task.exitstatus = 137 result in the your h_vmem being 137 bytes?

https://github.com/nextflow-io/nextflow/issues/332

probably solves your issue.  

--
  -  Alan Hoyle  -  al...@alanhoyle.com  -  http://www.alanhoyle.com/  -

[Sayantan Das]

Hi, Yes, I have using that, but that seems to work only inside a process. I am trying to make it look cleaner by moving it to the config files for certain labels. 

Sorry, I should have given more details. Yes, I have seen that issue and that is why I have been using clusterOptions to get the right memory. 

I initially had the following in my config file and it was throwing an error as well. Then I replaced the task.memory with task.exitStatus since errorStrategy was not throwing any error. So, yes, I am actually NOT trying to use the exitStatus for the memory, that was a debugging step and forgot to replace the task.exitStatus with task.memory.  It throws similar error has shown below:

Actual Snippet:

clusterOptions      = "-l h_vmem="+task.memory.toString().replaceAll(/[\sB]/,'')

Error:

Unknown config attribute `process.withLabel:sge_label.task.memory ` -- check config file: /home/sayantand/sci/analyses/hla_imputation_v2/src/nf/nextflow.config


In, summary I am trying to allow clusterOptions to update dynamically but define the formulas in the config file and NOT in the main.nf file. Hope that makes sense ?

On Tuesday, June 9, 2020 at 10:26:45 AM UTC-7, Alan H wrote:

As an aside, I know you can manipulate the memory request based on the task.attempt.  Here is an example derived from something we do:

process task_that_occasionally_needs_more_ram {

    cpus 8

    memory {16.GB * task.attempt }

    errorStrategy {task.attempt < 4 ? 'retry' : 'finish }

    maxRetries 4

    

    input: [...]
}

We let nextflow handle the details on how the memory allocations are specified for the executor.  We use SLURM primarily, but we used to use SGE before we moved to Nextflow.  

Also, wouldn't task.exitstatus = 137 result in the your h_vmem being 137 bytes?

https://github.com/nextflow-io/nextflow/issues/332

probably solves your issue.  

--
  -  Alan Hoyle  -  al...@alanhoyle.com  -  http://www.alanhoyle.com/  -

On Tue, Jun 9, 2020 at 8:33 AM Alan Hoyle <alan...@gmail.com> wrote:

Is there a reason you’re using clusterOptions to specify h_vmem rather than using process.memory and letting Nextflow figure out which options to send the executor?

--

Alan Hoyle - al...@alanhoyle.com - alanhoyle.com

---

From: 'Sayantan Das' via Nextflow <next...@googlegroups.com>
Sent: Tuesday, June 9, 2020 3:43:15 AM
To: Nextflow <next...@googlegroups.com>
Subject: Simple question on Dynamic resource allocation in config file

 

Hi, I am not sure if I am doing something silly, but it seems that we can dynamically assign the attribute `errorStrategy` but we cannot do the same for `clusterOptions`. The following is throwing an error. Any help would be appreciate. It runs fine if I highlight the clusterOptions

process {

    withLabel: sge_label {

        errorStrategy       = { task.exitStatus in 137..140 ? 'retry' : 'terminate' }

        clusterOptions      = "-l h_vmem="+task.exitStatus.toString().replaceAll(/[\sB]/,'')

        maxRetries          = 3

    }

}

ERROR: 

N E X T F L O W  ~  version 20.01.0

Launching `/home/sayantand/sci/analyses/hla_imputation_v2/src/nf/main.nf` [insane_sanger] - revision: 63e27b4ec0

Unknown config attribute `process.withLabel:sge_label.task.exitStatus` -- check config file: /home/sayantand/sci/analyses/hla_imputation_v2/src/nf/nextflow.config

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.

To unsubscribe from this group and stop receiving emails from it, send an email to next...@googlegroups.com.

[Sayantan Das]

I found a solution on gitter... I needed to use closures as follows:

clusterOptions      = {"-l h_vmem="+task.memory.toString().replaceAll(/[\sB]/,'')}

[Paolo Di Tommaso]

That's can be even simplified to: 

```

clusterOptions      = {"-l h_vmem=${task.memory.bytes}'')}

```

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/acd7244c-c122-4030-a5be-fdf3f0ddcb1fo%40googlegroups.com.

[Sayantan Das]

Beautiful. Thanks so much Paolo. 

You received this message because you are subscribed to a topic in the Google Groups "Nextflow" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nextflow/mS5Lw4meg6c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nextflow+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/CADgKzdxji4aYsd5eR3m87QL9aAWXPiooq3P_j%2BwckUJCpyq8_Q%40mail.gmail.com.