Memory issues with IQ-TREE MPI for large phylogenomic dataset

[Ava Ghezelayagh]

Hello!

I've been trying to build a species tree with the MPI version of IQ-Tree for 1109 taxa using a phylogenomic dataset (279 partitions, GTR model). I've been running iqtree-mpi single threaded for each MPI worker / slurm task. I'm trying to run this on a HPC cluster that uses cgroups to limit CPU and memory usage, but I can't get past the step of initializing the candidate tree set with the total amount of memory I have available (~5100 G of RAM that I've tried spreading out over a various number of cores (50-100)). The furthest I've gotten (with checkpointing) is 181 iterations, and no further progress is being made. 

Many of my jobs have quit due to insufficient memory allocation, so it appears IQ-Tree is significantly underestimating the memory needed for a single core, unless there is additional overhead that isn't warned about. I've tried using anywhere from 50-100G, depending on the number of cores I've requested. I've taken a look at some of the posts regarding running IQ-Tree with some very large datasets, and I've incorporated some of the suggestions (namely I added the following to my script: -safe -ninit 1 -pers 0.1), but if anything those additional options seem to have slowed things down further (or possibly even halted the program's progress entirely?). I know the student working on the MPI version has stopped doing so, but do you have any suggestions for how I might proceed to get this run actually moving along? 

My commands are as follows:

srun iqtree-mpi -s 75p_alignments.phylip -spp partition_file.txt -bb 1000 -bnni -st DNA -pre 75p_alignment -nt 1 -safe -ninit 1 -pers 0.1

Thank you so much for your help,

Ava

[Minh Bui]

Hi Ava,

I have improved the MPI version, so it should generally work. I’ll explain below why it takes so much RAM, but my suggestion: Use the multicore version together with MPI. This will save a lot of RAM. Say for example, if you have 10 nodes, each with 16 cores, then you can do:

mpirun -np 10 iqtree-mpi -nt 16 … [remaining options here]

This should more efficiently utilise all 160 cores using less RAM. The only option I didn’t test is: I don’t know if -bnni works with MPI version…

Now the explanation: The multicore version is shared memory, whereas MPI is not. Therefore, if a single run needs 10 GB RAM, then the above MPI-multicore run will need 10*10 GB in total for all 10 nodes. However, doing:

mpirun -np 160 iqtree-mpi -nt 1 …

will take 10*160 GB RAM. This is a factor of 16 times higher than previous run.

Let me know how this configuration works.

Cheers,

Minh

-- 
You received this message because you are subscribed to the Google Groups "IQ-TREE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iqtree+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/iqtree/29b4dbfe-bc6f-4fbd-96af-01ea49ef6302%40googlegroups.com.