Locking: Lustre and BeeGFS

[Andreas Wilm]

Dear all,

could anyone point me to a explanation of how file locking works in Nextflow? And more specifically: what mount options are required for Lustre and BeeGFS? 

For Lustre, is `-o flock` mandatory (see also this thread)? The Lustre Wiki states that "cluster-wide locking will have a detrimental impact on file system performance, and should only be enabled when absolutely required." Depending on how it works in Nextflow, would the apparently faster `localflock` be an alternative option, assuming the Nextflow control job is only ever run on one and the same node? The Lustre cluster we have access to is not managed by us, so I can't just try and would like to ask for the most minimally invasive change. 

Many thanks,
Andreas

[Paolo Di Tommaso]

NF uses an no lock  approach for processing tasks (since each task uses its own work directory there's no need for locks). 

A file system lock in only used by the embedded LevelDB database file that used to store the executed tasks metadata. This is store in the launching directory, this mean that not necessarily needs to be the shared file storage. It's enough that you launch the NF workflow in a local file system specifying the shared file system work directory with the `-w` option. 

Let me know if this answer your question. 

Cheers,
Paolo

 

--
You received this message because you are subscribed to the Google Groups "Nextflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/nextflow.
For more options, visit https://groups.google.com/d/optout.

[Andreas Wilm]

Yes it does. Excellent Paolo. Thank you very much! 

Andreas

To unsubscribe from this group and stop receiving emails from it, send an email to nextflow+u...@googlegroups.com.