Issue with Refinement Job Resource Utilization

[Milad Reyhani]

Dear team,

I am experiencing an issue with refinement jobs not fully utilizing the requested computational resources. I have also noticed that some of my colleagues are encountering similar behavior.

When I submit a job to the cluster and request, for example, 128 cores, the initial iterations run quickly. However, after the first few iterations, the process slows down significantly. I have attached a screenshot from a previous job in which I gathered metadata and ran the p2t2ptrd refinement. In that run, the first two positional refinements were completed within a day with full CPU usage, but resource utilization then dropped considerably, and subsequent iterations took much longer to complete. I have also attached the parameters used.

I would appreciate any advice on whether there is a way to improve or stabilize resource utilization during these later stages.

I look forward to your thoughts.

Best regards,
Milad

[Steve Ludtke]

Hi Milad,

The quick answer is that the biggest factor is probably storage bandwidth. For more details:

Debugging cluster performance issues can be very tricky. Not only does it depend on the characteristics of the data, but also the configuration of the cluster. Some EMAN2 tasks are limited to threaded (single node) parallelism, whereas others can make use of multiple nodes. CryoEM/ET image processing is extremely data-intensive, and in many cases, it isn't constrained by the CPU, but rather by data I/O. Some clusters are configured with high performance integrated storage systems which can reliably provide >1 GB/s of data read capability on each node simultaneously, but on many clusters there is a total data bandwidth available on the entire cluster and this is shared across nodes, so if you start a job running on a few nodes on a large cluster when the cluster is idle, it may perform very well, but then someone starts another big job on other nodes, and suddenly your data bandwidth drops.  This is just one potential situation.

In my lab, we've largely shifted away from using clusters to using individual workstations with (typ) 64 cores, very high performance M.2 storage, 10 Gb to a shared RAID with good performance, and 1-2 good GPUs.  When possible, we shift the entire project to the fast local M.2 storage, then run the refinements there using all of the cores on the workstation (and GPU for appropriate tasks).

So, we may be able to say more, but would need a lot more information:

- cluster configuration

- CPUs on a single node (specific model and count)

- RAM on a single node

- scratch availability on a single node (type of drive/speed)

- type of storage and speed where the project folder lives (the folder where you run eman2 commands from)

- project info

- box size of extracted subtilt stacks (often 2x the final box size)

- number of 3-D particles in the project

- number of tilts in a tomogram (typical if it varies)

- imposed symmetry (if any)

I'm not sure where in your plot the final "rd" stages came in, but r in particular can be very slow and rarely provides much of an improvement. Personally I don't usually use it.

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/eman2/50eec42c-decc-4a9e-ae26-ef833df049f4n%40googlegroups.com.
<Screenshot from 2026-05-04 14-12-54.png><0_spt_params.json>

[Muyuan Chen]

For cluster usage, also make sure you write scratch file to a location with enough storage. Specify --parallel thread:128:/path/to/scratch/ for this.

Also if you are on a new enough version, there is now a 'g' iteration that can replace 't'. It should be much faster and sometimes gets better results especially when there are many particles per tomogram. Maybe alternating p and g is better but I am still testing. 

Muyuan

To view this discussion visit https://groups.google.com/d/msgid/eman2/23DC6B44-F6C7-4DAF-93E7-BE8F54562B49%40gmail.com.

[Milad Reyhani]

Dear Steve and Muyuan,

Thank you very much for the detailed explanations.

I assume the next step would be to request allocation in the scratch directory from the cluster staff, as building a local machine capable of replacing the cluster is not feasible for us at this stage.

To give you a clearer picture of the setup and my workload, I run refinement jobs on CPU-only partitions (no GPUs) with the following specifications:

Nodes: 78 and 26
Cores per node: 128
Memory per node: ~977 GB and ~2 TB
Processor: Intel Xeon Gold 6448H
Max wall time: 30 days

Regarding the project, I am refining different regions of an icosahedral virus. The largest box size I currently use is 768 pixels (pixel size 1.603 Å/px), while for smaller regions it goes down to 96 pixels. I initially applied icosahedral symmetry for the whole virus and C5 symmetry for pentons, but individual pentons and other regions are now refined in C1.

Each tilt series should contain ~35 tilts, although I am not certain whether some are excluded during alignment and reconstruction. I have approximately 1,300 virus particles, which corresponds to ~15,600 pentons (box size 768), and ~78,000 monomers (box size 96). A cylindrical mask is applied around regions of interest.

The graph I shared corresponds to a job that did not complete the “d” iteration due to the time limit (20 days). The final peaks likely represent reconstruction steps from the “r” iteration or the beginning of the “d” iteration.

Thank you for pointing out the “g” iteration, we were not aware of it, and I will definitely test it.

Regarding scratch space, I know it is an NVMe-based filesystem. I will contact the cluster team for more details and to request allocation, but I would first need to provide an estimate of the required storage. I would appreciate your guidance on this.

Thank you again for all your help.

Best regards,
Milad

[Steve Ludtke]

Hi Milad, 

is the 768 box size the size with the 2x padding on the subtilt extraction or the unpadded size?  You're using a box size of 768 for the pentons, not the whole capsid?

That processor could be configured as 2 processors/node (64 cores, 128 threads) or 4 processors/node (128 cores, 256 threads). However, "threads" is deceptive, as it is not true additional processing power, though Intel tries to market it that way. When specifying a number of threads and/or MPI cores in EMAN2 jobs, limit yourself to the number of cores, not the number of hyperthreads.

Normally you shouldn't need to "request" scratch space, as the entire point of scratch space is exactly that; space which is used while a job is running and can then be discarded. Normally if you have a node allocated to you, you can use the entire local scratch space on that node while it's allocated. The question is whether you have true per-node local scratch available or the scratch is via some sort of shared scratch filesystem. If it is a shared scratch filesystem, then performance will depend on a lot of things (total storage bandwidth, shared filesystem used, interconnect speed, how heavily it's being used at a given moment in time).  Again, the key issue if you're trying to optimize processing is the actual (not theoretical max) storage bandwidth.

I can't give you much guidance on how much scratch will be used. I don't have enough details about your jobs. You can check yourself, though, if you have a job running. The default folder for scratch files if you didn't specify one should be /tmp. Just check the disk usage for your job in /tmp at different points of time during the job.

To view this discussion visit https://groups.google.com/d/msgid/eman2/f0a2d7e2-c8a7-4e27-972d-591a0d25e3c2n%40googlegroups.com.

[Milad Reyhani]

Hi Steve,

768 box size is with the 2x padding.

For using scratch space which I know is shared across multiple nodes, they request: "If you wish to use /data/scratch, please submit a request for a scratch filesystem directory."

That said, there is another space in the cluster, called the Local temp space. On each of the physical nodes there is a fast NVMe PCI-E card which will provide the fastest filesystem for the jobs. The normal capacity per node is 1.8TB, which is shared between all jobs running on that node. I can use it by writing to /tmp on each node. /tmp is local to each job and each worker node. But it is automatically cleaned once the job has finished. Is that fine? For this, I don't need to request for access.

[Steve Ludtke]

Yes, a local NVMe drive is ideal for scratch space, and /tmp is the default location where such files will be stored if you don't specify. So, for the --parallel option you should be fine with no changes.

Performance-wise, the type of storage where your project folder lives when you run your jobs is critical. It sounds like they probably provide a generic long-term storage filesystem to everyone, but likely that filesystem has limited performance. The "scratch" filesystem probably has higher bandwidth, but I'm just guessing based on typical cluster configurations. So, you have 3 "tiers" of storage:

1) main storage, probably a very large RAID array or somesuch, but with limited bandwidth to the cluster nodes

2) high performance shared scratch, a much smaller RAID or distributed storage array with high bandwidth to the compute nodes, for temporary use while jobs are running

3) local scratch (separate on each node). Highest possible bandwidth, but very limited capacity and lifetime

On such a cluster, the optimal way to run an EMAN2 job:

If you are running on multiple nodes with --parallel=mpi and --thread:

- at the beginning of the job, clone your entire project folder to the shared scratch drive (or at minimum info/, particles/, particles3d/, sets/, .eman2log.txt, .eman3log.txt )

- cd to the project folder on the shared scratch drive

- run your e2spt_refine... command, point the scratch drive for --parallel=mpi to the node-local scratch space (/tmp in your case)

- at the end of the job, rsync the project back to its permanent location

- if you aren't immediately going to run another job, remove the project folder from shared scratch

If you are running on a single node:

- at the beginning of the job, if you have enough space, clone the entire project to the node-local scratch space (if there isn't enough space, you can use the shared scratch instead)

- cd to the project folder on the node-local scratch space 

- run your e2spt_refine... command, point the scratch drive for --parallel=mpi to the node-local scratch space (/tmp in your case)

- at the end of the job, rsync the project back to its permanent location (critical, since /tmp may get automatically cleaned up at the end of the job)

- to be polite, in case the auto-cleanup features aren't working well, removing the project folder from the local scratch is a good idea

While it _can_ work to only copy the necessary files as mentioned above (  info/, particles/, particles3d/, sets/, .eman2log.txt, .eman3log.txt ), note that this means your refinement folder name will be spt_00 not a higher number, and if you then rsync back to permanent storage it will overwrite your previous spt_00 results, so you, instead would rsync the spt_00 file to the correct (next sequential number) name on permanent storage. In general I recommend avoiding this approach and rsyncing the entire project folder in both directions, as the chances of making a critical mistake are greatly reduced.

If you're using MPI on a cluster like this, your job would run faster on 4 nodes than on 2 nodes, but it will NOT be faster by a factor of 2. There are various things which can limit the total performance of a job on a large number of processes (storage and inter-process communication being big ones). When you start seeing a job consistently failing to use all of the CPU time available, this usually means that you have saturated some other cluster resource, and you may actually get better total performance using fewer nodes in such situations. Indeed, there are significant speedups you can gain by running on a single node. If your nodes are actually configured with 128 physical cores, you may actually get faster overall results running on a single node with --parallel=thread:128:/tmp and --threads=128 than you would running on, say, 2 nodes with MPI, but it's possible that running on 2 will still get your results faster (but not 2x faster). You can read about Amdahl's law if you want to understand some of the issues, but it isn't the only factor. 

To view this discussion visit https://groups.google.com/d/msgid/eman2/0352742e-5b89-43f4-a8f3-fa5fcc406782n%40googlegroups.com.

[Milad Reyhani]

Hi Steve,

Thank you for the excellent explanation. It really helped me understand the situation much better.

I will try moving the entire project folder to the scratch drive and run the project from there.

[Milad Reyhani]

Hi Steve and Muyuan,

Wanted to follow up on this thread with what we eventually found, since the root cause turned out to be more specific than general storage bandwidth, and I think it is worth sharing in case others hit the same wall.

Working with our cluster's HPC support team, we found the actual bottleneck was not raw read or write throughput. It was the .lst index files specifically. Every particle lookup opens, reads and seeks these small text files, and across thousands of particles, many iterations, and 128 concurrent threads, that adds up to an enormous number of tiny, latency sensitive operations. On a shared or networked filesystem, each of those tiny operations is a network round trip, and 128 threads hammering the same small file at once creates metadata contention that has almost nothing to do with bandwidth in the usual sense. This showed up clearly using lsof on the running job, which showed the vast majority of file activity going to these tiny .lst files rather than the actual particle data.

Moving the whole project to local scratch, as you suggested, is the right fix, but there is a gotcha worth knowing about if you try it the simple way. If you only move the working or output folder (spt_XX or similar) to scratch and symlink it back into place, while your shell's current working directory stays on the permanent filesystem, EMAN2's internal file rewriting (used for example when compressing hdf output) breaks with an Invalid cross device link error. That is because it relies on os.rename, which can only move files within a single filesystem, and the temp filename it generates is relative to the current working directory, not the target folder.

The robust fix is to make your shell's actual working directory live on local scratch too, not just the output subfolder. Concretely:

At the start of the job, create a working directory on local scratch.
Symlink every top level item in your project (particles, particles3d, masks, sets, info, other run folders) into that scratch directory, pointing back at the permanent filesystem. These stay as symlinks, not copies, since bulk particle data is often far bigger than available local scratch.
Copy (not symlink) the actual working or output folder for this run into the scratch directory as a real local copy.
Cd into the scratch directory before running any e2spt commands.
At the end of the job, rsync the working or output folder back to permanent storage. For long jobs, an hourly rsync checkpoint back to permanent storage is cheap insurance against losing progress if the job dies.

Since EMAN2's convention is to always run from a fixed top level project directory and never cd into subfolders, this works cleanly. All relative paths resolve the same way whether a given subfolder is a real local directory or a symlink back to permanent storage.

This should generalize to any cluster with per node local scratch (NVMe or otherwise) and a shared or networked permanent filesystem, regardless of the specific storage technology involved. The key idea is just: keep your current working directory and your active output folder on the same filesystem as each other, symlink whatever is too big to copy, and sync back at the end.

We also ran with LD_PRELOAD set to a jemalloc library, on our HPC team's suggestion, as a general fix for malloc contention in heavily multithreaded jobs. Worth mentioning, though we always ran it together with the storage fix, so we honestly cannot say how much it contributed on its own.

Thanks again for the pointers that got us started on this. Hope it helps if anyone else runs into the same wall.

Milad

[Steve Ludtke]

Thanks for following up with your outcome. I'm curious what shared filesystem is being used on your cluster?  

For the read-only access which is normally going to be happening in this sort of processing, with a filesystem like NFS there should be local caching of the file, and the cache would be shared across threads, so the only time a network roundtrip should be required is for periodically making sure the file is unchanged on the server, which in typical configurations would be at least a few seconds?  

To view this discussion visit https://groups.google.com/d/msgid/eman2/5b1591be-9314-432b-bf98-aaa63054fc6an%40googlegroups.com.

[Milad Reyhani]

Hi Steve,

We're on GPFS (IBM Spectrum Scale), not NFS, should've mentioned that originally.

Not 100% sure on the exact mechanism, but my guess is it comes down to the access pattern. EMAN2's .lst access doesn't hold one file handle open and read from it. It opens and closes the file for every single particle lookup. GPFS uses token based locking to keep things consistent across concurrent access, and my guess is each fresh open triggers a token renegotiation, especially with 128 processes hammering the same file. That's pretty different from NFS's close to open caching, which mostly helps when you keep a handle open, not when you're constantly reopening it.

If that's right, this problem might just not show up the same way on NFS, given the caching model you described.

But this is at the edge of what I actually know about GPFS internals though. Happy to ask our HPC guy if you want a more solid answer.

Milad

[Steve Ludtke]

Ahh, that explains it. Yes, GPFS has very different properties than NFS. NFS does have a metadata caching mechanism which can span multiple open/close cycles (there is a parameter which sets a time limit for how long it should "trust" the metadata from the last cache). Even when this number is small (a few seconds) it can prevent the sort of "thrashing" behavior we're discussing.

Having said that, the way the LSXFile mechanism was designed is supposed to explicitly avoid many open/close operations on the .lst file. Under designed use, the LSXFile would be opened once and reused, so the individual referenced image files may be opened and closed a lot, but the .lst file itself would not. However, if the program is designed to work with any sort of input file, not just .lst files, it will treat the .lst file as an image file. This works, ie - EMData.read_image() will treat .lst files as actual image files, not just metadata files, but is not very efficient because the file gets opened and closed a lot. So, in the end it depends a lot on the specific EMAN2 program you're using, and whether it requires the use of .lst files for input.