Speed up "gather-pvalues-for-each-gene"

[Ashley Qin]

Hi Peter, is there any way to speed up the processing time for the script "gather-pvalues-for-each-gene"? I have almost 900 phenotypes to process, which translates to about 20,000 tasks. I'm using the SLURM cluster scheduler to handle this task, but even after 4 days of running the job reaches the time limit. 

[pjvandehaar]

Sorry about getting back to you so late.  Presumably you've figured it out by now.

Yeah, that step is too slow.  It could run 4x faster with a better approach, and I wrote out how to do it at the top of https://github.com/statgen/pheweb/blob/master/pheweb/load/gather_pvalues_for_each_gene.py .  I'll get to it eventually, but if you'd like to work on it I will help.