Cannot finish run in time

[Xuan]

Hi there,

Thanks for providing such a cool tool! However, I have tried a few times to get this running with the data provided below. But it takes me forever - like literal forever, to complete the run. Even with reduced number of 'run'. I am wondering if there's any way I can speed up the test? I have 23 taxon and testing networks from 0 to 10. All gene trees were 10% nodes collapsed and rerooted, as well as the species astral tree. 

Is it possible I can use multi-core to do this? or you may have any suggestions?

Many thanks!

Weixuan

[Cécile Ané]

Yes absolutely, you can and should use multiple cores if you are running this on a cluster. There is documentation on how to do this here.

It looks like your scripts (julia script and slurm submission script) are already well setup to use multiple cores.

In short: slurm should request 10 CPUs per task, if we want SNaQ to do 10 independent runs each time. Otherwise, when we ask Julia to use 10 "workers", all workers will use the same CPU: they will have to share the same CPU if slurm didn't give julia 10 separate CPUs for the same julia session. For 3 runs, we only need to request 3 CPUs. For 10 runs, we need 10 CPUs for the total computation time to be similar to the time of 1 single run.

It looks like your slurm submit script requests 30, which is quite more than what you need. Perhaps that puts your job down in the priority list?

In your slurm submit file, I'm worried about the variable "$SLURM_ARRAY_TASK_ID_5runs". I wonder if this will cause an error at the end of snaq, to redirect its output. Should it be ${SLURM_ARRAY_TASK_ID}_5runs perhaps? I see that this error in 

[Xuan]

Thanks! for each 'run' which can only take 1 CPU, and I can change that, correct? I really hope to speed up this bit. I will reduce the CPU to 5, since I will do only maximum 3 replicates per hybridization event. Sorry, this is sounds bit confusing... how long you would recommend to the set the time to?

I will change the output too!

Many thanks,

Weixuan

[Cécile Ané]

If you have access to a cluster, doing 30 runs or 3 runs should take about the same time... I don't really see the advantage to decreasing the number of runs, given that you can parallelize them. It only decreases the chances of finding the network with the best score.

To reduce time, the best strategy would be subsample your taxa: reduce the number of taxa from 23, to say, 15 (arbitrarily). If there is a clade that are very well supported, then you could sample only 2 taxa from this clade, if introgression within such a clade if not of primary interest. Or you could take several subsamples: one in clade A to look at introgression within clade A, another subsample of taxa in clade B to look at introgression within clade B, and a higher-order taxon subsampling with 2 taxa from A and 2 from B, etc., to look at introgression between these clades, perhaps more ancient. Reducing the number of taxa in each analysis is really what will work best to reduce computation time.