Adjust memory request based on file size?

[Alan Hoyle]

I want to be able to adjust the memory allocation based on the file size of one of the inputs.  (e.g. the size of the STAR index in GB + 3GB, since the full index needs to be in RAM.)

I see an older  discussion here https://github.com/nf-core/deepvariant/issues/5 and https://github.com/nextflow-io/nextflow/issues/922, but I don't see a good resolution suggested.  

--
  -  Alan Hoyle  -  al...@alanhoyle.com  -  http://www.alanhoyle.com/  -

[Luc Dehaspe]

Hi Alan,

Just had to solve the same problem. This worked for me to use a disk that is 4 times larger than input "my_file" stored in a cloud bucket:

process my_proc {

...

  disk { 4.B * my_file.size() }

....

input:

   path(my_file)

...

}

Kind regards,

--luc

Op di 8 jun. 2021 16:30 schreef Alan Hoyle <al...@alanhoyle.com>:

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/CAKxVof0s-8OHhc66PbqFkYdzakaCcYkFiT6hfAHD%3DpSyuXkphQ%40mail.gmail.com.

[Paolo Di Tommaso]

This also may help 

http://nextflow-io.github.io/patterns/index.html#_conditional_resources_definition

p

To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/CAGHwEgxxB6U_2HPbzAa7k8GNrskbhq%2BXh6CaHvu4t%2BYntRfYYg%40mail.gmail.com.

[Alan Hoyle]

Thanks all Paolo and Luc!  We'll see what we can do here.  

--
  -  Alan Hoyle  -  al...@alanhoyle.com  -  http://www.alanhoyle.com/  -

To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/CADgKzdwbqeG%3DqfoDX_QFWvkScKYf0tFVWVT4WsW93P2Mc8mpgw%40mail.gmail.com.

[Kemin Zhou]

I have been using the dynamic resource definition method with the binary operator { file.size() < CUTOFF ? choice1 : choice2 }

and have never tried to use more fine scale functions say using a custom groovy functions

memory { memneed = computeMemoryNeeded(file.size()); return memneed.GB }

The computeMemoryNeeded() function is defined globally in the nextflow script.

I am not sure such method will work or not.  Theoretically, any method should work inside the closure.

Would like to get some comments and direction from you

Thanks for all the good work.

To view this discussion on the web visit https://groups.google.com/d/msgid/nextflow/CADgKzdwbqeG%3DqfoDX_QFWvkScKYf0tFVWVT4WsW93P2Mc8mpgw%40mail.gmail.com.

--

Kemin Zhou

858 366 8260