Scratch directory and cluster question

[Doulin Shepherd]

Hello EMAN2 team,

I have a couple of questions about how to run EMAN2 most efficiently on our HPC. I know that HPC clusters configs are variable but I hope these questions are general enought that some advice can be given.

I've been looking at the documentation and see that EMAN2 has the option to write files to a scratch directory. Our HPC setup allows for two types of scratch directories: 1) a local NVMe drive on the compute node, and 2) a separate, shared NVMe file system. For both options I need to provide an estimate of the storage needed.

Normally, I would use option 1 because it's the fastest. However, it has limited space, so I wanted to clarify something. When the --parallel=thread:<n>:<tmp_path> option is used, are the required particles copied to the temporary path, or only the intermediate files? Also, is there anyway to estimate the file size of the intermediate files?

This brings me to a related question: since I'm not specifying a scratch directory, the default location is /tmp, which is mapped to RAM on our HPC (unless I use option 1). Some of my runs require me to request up to 1 TB of RAM. Is this because all the large intermediate files are being stored in memory? If so, would specifying a scratch directory reduce the RAM requirements?

Thank you for your help.

[Steve Ludtke]

Hi Doulin,

it depends a lot on the specific program you're running and the data sets you're using. Each program using --parallel decides how much to rely on existing data files, and how much to put in the per-thread files in the temporary folder. I normally use a local NVME drive for this purpose so I don't need to worry about it, but if the NVME drive is small, this might potentially still be a problem. Most EMAN programs use this cache only for relatively small metadata files, but I think one of Muyuan's pipelines (I don't recall which, maybe he can comment) actually caches some large chunks of data this way?

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/eman2/eae3d682-9c62-45d9-80d2-bb6c8a9e1b01n%40googlegroups.com.

[Doulin Shepherd]

Hi Steve,

I'll be using the subtomogram averaging pipeline, specifically e2spt_refine_new.py. We've recently started working with a large dataset, containing about 80,000 particles. I typically run refinements on this dataset with --parallel=thread:72 --threads=72, which usually requires just under 1 TB of memory.

I know that Relion and CryoSPARC copy the data to a scratch path before running, which has no impact on memory usage, but I assume the EMAN2 scratch function works differently. Currently, the particles and particles3d folders are on "slow" drives.

I'm trying to determine the most efficient method to handle this in terms of RAM requirements and processing time but I am quite new to using EMAN2 on the cluster. 

(A) Setting the scratch directory as NVMe drive partion on the cluster.

(B) Setting the scratch directory to the node's NVMe drives (which have limited space).

(C) Copying the necessary folders to the NVMe drive partition on the cluster, running from there and then copy the results back to the original project directory.

I'm not sure which of these is the most effective approach 

Cheers,
Doulin 

[Muyuan Chen]

I think most programs in the e2spt_xxxx pipeline that use --parallel read/write only small scratch files, i.e. only passing image indices and return transform of aligned particles per thread. There were some old programs that write image data to scratch that but I assume they all died during the python version transition since I never upgraded them anyway...

Particle files are not copied to scratch automatically. I suppose it can be much faster if you copy them to a fast drive before processing. I typically just run on slow drives. I don't have that much space on fast drives and I am quite good at waiting... 

--

--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/eman2/9797c660-fe9b-443d-b471-4e4da53dd056n%40googlegroups.com.