Voila modulize freezing in some cases

[ЖИРАНДОЛЬ]

Hi everyone! 

I'm having some issues running voila (version 2.4.dev3+g85d07819) modulize mode.
I've successfully runed majiq build(version 2.4.dev3+g85d07819) , majiq deltapsi and get corect *.voila and *.tsv files. Commands were like:

deltapsi:
```majiq deltapsi -j 64 -o dpsi/ --minreads 5 -grp1 build/x1.fq.majiq build/x2.fq.majiq build/x3.fq.majiq build/x4.fq.majiq -grp2 build/x8.fq.majiq build/x5.fq.majiq build/x6.fq.majiq build/x7.fq.majiq -n E4_WT_KO_ E4_KO_hom```
modulize:
voila modulize -j 32 -d categorize/E4_WT_KO_-E4_KO_hom build/splicegraph.sql E4_WT_KO_-E4_KO_hom.deltapsi.voila --changing-between-group-dpsi 0.01 --non-changing-between-group-dpsi 0.01

i ran some comparisons in parallel, like cat batch_cmds | parallel 'eval {...}' (but, anyway, running commands not in parallel give me the same result)

But, in some cases, voila just freezes and when some batch commands are freezing in stage Processing Genes and Modules [55136/55359], and I got no data like cassette.tsv and other ouput of voila modulize.

last line of log file looks like:
2022-05-24 14:26:20,574 (PID:764828) - INFO - Writing TSVs to /path/to/folder

can you help me how to solve this issue? 


Best wishes,
Alex

[Paul Jewell]

Hi Alex,

This is very odd, and it a new issue, so you may need to help me with some debugging and explanation.

-Can you describe the environment and run conditions? Laptop or HPC, running over ssh? in a screen session? using a grid manager? locally?  What kind of machine and storage setup are you running on?

-When the command freezes, is their any activity still going on with the process? On Linux, for example, you can run the command '$ top' and check if there is still significant disk / CPU usage by the voila process at that time.

-Can you try a smaller run,  with and without multithreading?

   -for example, $ voila modulize -j 1 --debug-num-genes 100
                           $ voila modulize -j 1 --debug --debug-num-genes 100
                           $ voila modulize -j 2 --debug-num-genes 100

See if any of these complete without the freezing issue.

- It may also be that something is corrupt with the splicegraph/voila file when it gets to gene 55136. Are you able to run something like $ sha1sum splicegraph.sql  to verify that there are no I/O errors with these files?

Thanks.

[Nikshep Grampurohit be21b025]

Hi,

Even I've been facing the same issue. I'm running voila modulize on 72 PSI files and it freezes in the middle. I've run it multiple times and it always freezes at a different gene number, it seems to be random. It is not using any CPU or disk storage when it freezes. 

I'm currently running it on an Azure VM with 32GB RAM and 128GB storage over SSH. 

[San Jewell]

Hello,

Were you able to run any of the test commands above to check if it's able to complete and whether multithreading has any effect on this? Also I would ask you check the memory usage if you are using multi threading, and check if it is close to running out when it freezes, as multiprocessing tends to behave strangely if the OS kills some threads for OOM before the main process is complete.

Let me know what you find,

-San

[Nikshep Grampurohit be21b025]

Hi San,

It runs for both of the above commands. It works well for a smaller number of samples (20-30) as well. I suspect it could be something relating to the RAM running out. Is there any way to limit the amount of RAM voila uses? Or any other solution.

Thank you

Nikshep

[San Jewell]

Hi Nikshep,

The memory should only grow mostly at the very beginning of the run, it may get slightly larger but should not continue growing significantly. However, if it is already close to the edge of what's available, that could pose a problem. As temporary solutions to check, I would say you could 1) add some temporary space to a swapfile as a safeguard 2) try running with a few fewer threads, as each thread in python needs to allocate its own memory space.

Additionally, I've noticed with some annotations / datasets / filter sets, the number of junctions can grow very high for a handful of complex genes, and this can cause the run time and memory use to grow while processing those few complex cases.

Let me know if these help.