how should I voila modulize with multiple samples

[jin jiacheng]

Hi San,

I hope this mail finds you well, I have encountered a problem when i'm trying to voila modulize with multiple input psicov files and sgc files, because i want to check the psi of every sample of mine in every lsv. My command was

$voila modulize -d voila /rna_seq/results/majiq/build/sg.zarr /rna_seq/results/majiq/psi/sample_1.psicov 

/rna_seq/results/majiq/psi/sample_2.psicov 

/rna_seq/results/majiq/sgc/sample_1.sgc 

/rna_seq/results/majiq/sgc/sample_2.sgc --show-per-sample-psi --show-read-counts -j 20 (I demonstrated with only 2 samples but actually there's more, and these samples have only one replicate in it) 

and i am using the latest version 3.0.18.

However, the data in two columns(sample_1.median_psi, sample_2.median_psi) of the output tsvs turned out to be the same in every row, and here i show the last six columns in the junctions.tsv, is there something wrong with my command and if so how should i revise? I have tried to add --psicov-grouping-file but it won't work, or is it because i have only one rep? or is there something wrong with my input file, could you please kindly help me out with this problem and explain the reason why? Please let me know if any additional information is needed.

sample_1.median_reads   sample_1.median_psi    sample_1.var_psi  

sample_2.median_reads   sample_2.median_psi    sample_2.var_psi 

347 8.971e-01 5.001e-04 291 8.971e-01 8.722e-04

 21 6.646e-02 4.465e-04 25 6.646e-02 7.517e-04

246 7.836e-01 1.539e-03 126 7.836e-01 1.982e-03

50 1.890e-01 1.461e-03 27 1.890e-01 1.904e-03

Thank you,

Jiacheng

[San Jewell]

Hi Jiacheng, 

You are correct. After a recent patch to address multiple groups inside single psi-coverage files, the wrong mode was specified, which caused the modulizer to take the mean over the file groups instead of showing each group individually. I have just pushed a patch which should address this. Please let me know if the problem persists after updating. 

Thank you for the report!

-San

[jin jiacheng]

Hi San,

Sorry for my late reply, the problem is successfully solved after the update! But the voila modulize process became very slow and it might take more than ten hours to finish with only two samples with 20 processes, is there any way to improve? Thanks again for your answer!

Jiacheng

[San Jewell]

Jiacheng, 

No worries. 

For your new question, I have compared and averaged some run times between the new and old versions, human transcriptome with three samples. In my attempt to reproduce your issue, I actually have seen a minor improvement in run times in the newer version. Is it possible that there is some other factor that has changed for you? 10 hours sounds exceedingly excessive. What does your computing environment look like? If we cannot determine from discussion, would you be willing to share some sample data demonstrating the issue so that I may see if I can reproduce it?

Thanks, 

-San

[jin jiacheng]

Hi San, 

Thanks for your answer, I can provide the sgc file and psicov file of two samples, and the sg.zarr for you. And I will show all of command to see if there is issue with my command. I figured that part of the reason was that the sg.zarr I used earlier was based on my whole dataset which has like hundreds of samples so I tried to build a sg.zarr based on only two samples and it had some improvement but still took about 50 minutes with 20 processes. So I also provided the files constructed based on two sample sg.zarr and whole dataset, which has 2 in its file name. The species were oryza sativa. If modulizing on such a big dataset was too time-consuming, is it appropriate to perform analysis just on psicov output tsv?

majiq-v3 build /share/home/jinjc/rna_seq/results/majiq/annotation/sg.zarr sg.zarr  --groups-tsv config.tsv

majiq psi-coverage sg.zarr Salt_NH247.psicov /share/home/jinjc/rna_seq/results/majiq/sj/Salt_NH247.sj -j 20 --overwrite 
majiq psi-coverage  sg.zarr Salt_NH246.psicov /share/home/jinjc/rna_seq/results/majiq/sj/Salt_NH246.sj   -j 20 --overwrite 
majiq-v3 sg-coverage  sg.zarr Salt_NH247.sgc /share/home/jinjc/rna_seq/results/majiq/sj/Salt_NH247.sj -j 20 --overwrite 
majiq-v3 sg-coverage  sg.zarr Salt_NH246.sgc /share/home/jinjc/rna_seq/results/majiq/sj/Salt_NH246.sj -j 20 --overwrite 
voila modulize -d voila sg.zarr /share/home/jinjc/rna_seq/scripts/Salt_NH247.psicov /share/home/jinjc/rna_seq/scripts/Salt_NH246.psicov /share/home/jinjc/rna_seq/scripts/Salt_NH247.sgc /share/home/jinjc/rna_seq/scripts/Salt_NH246.sgc --debug --show-per-sample-psi --show-read-counts -j 20 --overwrite

I was computing on HPC, my environment was 

$ conda list
# packages in environment at /share/home/jinjc/miniconda3/envs/majiq:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
asciitree                 0.3.3                    pypi_0    pypi
binutils_impl_linux-64    2.36.1               h193b22a_2    conda-forge
binutils_linux-64         2.36                hf3e587d_10    conda-forge
blinker                   1.9.0                    pypi_0    pypi
bokeh                     3.8.1                    pypi_0    pypi
bottleneck                1.4.0                    pypi_0    pypi
bzip2                     1.0.8                hda65f42_8    conda-forge
c-ares                    1.34.6               hb03c661_0    conda-forge
ca-certificates           2025.12.2            h06a4308_0  
cachelib                  0.13.0                   pypi_0    pypi
certifi                   2025.11.12               pypi_0    pypi
cffi                      2.0.0                    pypi_0    pypi
charset-normalizer        3.4.4                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
cloudpickle               3.1.2                    pypi_0    pypi
cmake                     3.26.4               hcfe8598_0    conda-forge
contourpy                 1.3.3                    pypi_0    pypi
cryptography              43.0.0                   pypi_0    pypi
dask                      2024.7.1                 pypi_0    pypi
dask-expr                 1.1.9                    pypi_0    pypi
distributed               2024.7.1                 pypi_0    pypi
exceptiongroup            1.3.1              pyhd8ed1ab_0    conda-forge
expat                     2.7.3                hecca717_0    conda-forge
fasteners                 0.20                     pypi_0    pypi
fastp                     0.23.4               h125f33a_5    bioconda
flask                     3.0.3                    pypi_0    pypi
flask-session             0.8.0                    pypi_0    pypi
flask-wtf                 1.2.1                    pypi_0    pypi
fsspec                    2025.12.0                pypi_0    pypi
gcc_impl_linux-64         11.2.0              h82a94d6_16    conda-forge
gcc_linux-64              11.2.0              h39a9532_10    conda-forge
gitdb                     4.0.12                   pypi_0    pypi
gitdb2                    4.0.2                    pypi_0    pypi
gitpython                 3.1.43                   pypi_0    pypi
gtfparse                  1.3.0                    pypi_0    pypi
gunicorn                  23.0.0                   pypi_0    pypi
gxx_impl_linux-64         11.2.0              h82a94d6_16    conda-forge
gxx_linux-64              11.2.0              hacbe6df_10    conda-forge
h5py                      3.11.0                   pypi_0    pypi
htslib                    1.18                 h81da01d_0    bioconda
idna                      3.11                     pypi_0    pypi
importlib-metadata        8.7.0              pyhe01879c_1    conda-forge
importlib-resources       6.5.2              pyhd8ed1ab_0    conda-forge
importlib_resources       6.5.2              pyhd8ed1ab_0    conda-forge
intervaltree              3.1.0                    pypi_0    pypi
isa-l                     2.31.1               hb9d3cd8_1    conda-forge
itsdangerous              2.2.0                    pypi_0    pypi
jinja2                    3.1.4                    pypi_0    pypi
kernel-headers_linux-64   5.14.0               he073ed8_3    conda-forge
keyutils                  1.6.3                hb9d3cd8_0    conda-forge
krb5                      1.21.3               h659f571_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libblas                   3.11.0          5_h4a7cf45_openblas    conda-forge
libcblas                  3.11.0          5_h0358290_openblas    conda-forge
libcurl                   8.8.0                hca28451_1    conda-forge
libdeflate                1.25                 h17f619e_0    conda-forge
libedit                   3.1.20250104    pl5321h7949ede_0    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libexpat                  2.7.3                hecca717_0    conda-forge
libffi                    3.5.2                h9ec8514_0    conda-forge
libgcc                    15.2.0              he0feb66_16    conda-forge
libgcc-devel_linux-64     11.2.0              h0952999_16    conda-forge
libgcc-ng                 15.2.0              h69a702a_16    conda-forge
libgfortran               15.2.0              h69a702a_16    conda-forge
libgfortran5              15.2.0              h68bc16d_16    conda-forge
libgomp                   15.2.0              he0feb66_16    conda-forge
liblapack                 3.11.0          5_h47877c9_openblas    conda-forge
liblzma                   5.8.1                hb9d3cd8_2    conda-forge
liblzma-devel             5.8.1                hb9d3cd8_2    conda-forge
libnghttp2                1.58.0               h47da74e_1    conda-forge
libnsl                    2.0.1                hb9d3cd8_1    conda-forge
libopenblas               0.3.30          pthreads_h94d23a6_4    conda-forge
libsanitizer              11.2.0              he4da1e4_16    conda-forge
libsqlite                 3.46.0               hde9e2c9_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx                 15.2.0              h934c35e_16    conda-forge
libstdcxx-devel_linux-64  11.2.0              h0952999_16    conda-forge
libstdcxx-ng              15.2.0              hdf11a46_16    conda-forge
libuuid                   2.41.2               h5347b49_1    conda-forge
libuv                     1.51.0               hb03c661_1    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libzlib                   1.2.13               h4ab18f5_6    conda-forge
locket                    1.0.0                    pypi_0    pypi
lz4                       4.4.5                    pypi_0    pypi
markupsafe                2.1.5                    pypi_0    pypi
msgpack                   1.1.2                    pypi_0    pypi
msgspec                   0.20.0                   pypi_0    pypi
narwhals                  2.14.0                   pypi_0    pypi
ncurses                   6.5                  h2d0b736_3    conda-forge
networkx                  3.3                      pypi_0    pypi
ninja                     1.13.2               h171cf75_0    conda-forge
numcodecs                 0.13.1                   pypi_0    pypi
numpy                     2.0.1                    pypi_0    pypi
openssl                   3.6.0                h26f9b46_0    conda-forge
packaging                 25.0               pyh29332c3_1    conda-forge
pandas                    2.2.2                    pypi_0    pypi
partd                     1.4.2                    pypi_0    pypi
pathspec                  0.12.1             pyhd8ed1ab_1    conda-forge
pillow                    12.0.0                   pypi_0    pypi
pip                       25.3               pyh8b19718_0    conda-forge
polars                    1.2.1                    pypi_0    pypi
psutil                    6.0.0                    pypi_0    pypi
pyarrow                   16.1.0                   pypi_0    pypi
pyarrow-hotfix            0.7                      pypi_0    pypi
pybind11                  2.11.1          py311h9547e67_2    conda-forge
pybind11-global           2.11.1          py311h9547e67_2    conda-forge
pycparser                 2.23                     pypi_0    pypi
python                    3.11.9          hb806964_0_cpython    conda-forge
python-dateutil           2.9.0.post0              pypi_0    pypi
python_abi                3.11                    8_cp311    conda-forge
pytz                      2025.2                   pypi_0    pypi
pyyaml                    6.0.3                    pypi_0    pypi
readline                  8.3                  h853b02a_0    conda-forge
requests                  2.32.3                   pypi_0    pypi
rhash                     1.4.3                hd590300_2    conda-forge
rna-majiq                 3.0.19.dev1+gde48918a5          pypi_0    pypi
rna-majiq-meta            3.0.0                    pypi_0    pypi
rna-moccasin              3.0.19.dev1+gde48918a5          pypi_0    pypi
rna-voila                 3.0.19.dev1+gde48918a5          pypi_0    pypi
scikit-build-core         0.11.6             pyh7e86bf3_1    conda-forge
scipy                     1.14.0                   pypi_0    pypi
setuptools                80.9.0             pyhff2d567_0    conda-forge
setuptools-scm            9.2.2              pyhd8ed1ab_0    conda-forge
setuptools_scm            9.2.2                hd8ed1ab_0    conda-forge
six                       1.16.0                   pypi_0    pypi
smmap                     5.0.2                    pypi_0    pypi
smmap2                    3.0.1                    pypi_0    pypi
sortedcontainers          2.4.0                    pypi_0    pypi
sysroot_linux-64          2.34                 h087de78_3    conda-forge
tblib                     3.2.2                    pypi_0    pypi
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tomli                     2.3.0              pyhcf101f3_0    conda-forge
toolz                     1.1.0                    pypi_0    pypi
tornado                   6.5.4                    pypi_0    pypi
tqdm                      4.66.4                   pypi_0    pypi
typing-extensions         4.15.0               h396c80c_0    conda-forge
typing_extensions         4.15.0             pyhcf101f3_0    conda-forge
tzdata                    2025.3                   pypi_0    pypi
urllib3                   2.6.2                    pypi_0    pypi
waitress                  3.0.0                    pypi_0    pypi
werkzeug                  3.0.3                    pypi_0    pypi
wheel                     0.45.1             pyhd8ed1ab_1    conda-forge
wtforms                   3.1.2                    pypi_0    pypi
xarray                    2024.6.0                 pypi_0    pypi
xyzservices               2025.11.0                pypi_0    pypi
xz                        5.8.1                hbcc6ac9_2    conda-forge
xz-gpl-tools              5.8.1                hbcc6ac9_2    conda-forge
xz-tools                  5.8.1                hb9d3cd8_2    conda-forge
zarr                      2.18.3                   pypi_0    pypi
zict                      3.0.0                    pypi_0    pypi
zipp                      3.23.0             pyhcf101f3_1    conda-forge
zlib                      1.2.13               h4ab18f5_6    conda-forge
zstd                      1.5.6                ha6fb4c9_0    conda-forge

Thanks,

Jiacheng

[jin jiacheng]

 jinjc_samples.rar

--
You received this message because you are subscribed to the Google Groups "Biociphers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to majiq_voila...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/majiq_voila/a62f270c-8030-4ab5-aa75-d9dfe1cae201n%40googlegroups.com.

[San Jewell]

Hi Jiacheng, 

I feel something very odd must be happening. As a test, I have run your samples on my local laptop, as the data size is small:

The run takes about 19 minutes on a single core, and about 5 minutes on 6 cores. 

What I am imagining may be happening is that there is significant contention (likely only temporary) on a storage subsystem of your HPC. As voila is I/O bound and makes a number of disk requests, this may have caused the slowdown. As a test, could you try to copy the data files to a scratch ssd on the compute node, or to /dev/shm on the compute node, before running voila on it (remember to delete the temporary files afterwards)? Could you take a screenshot running the time command as I have done? 

Thank you, 

-San

[jin jiacheng]

Hi San,

Thanks for your advice, I've tried to run in /dev/shm but the problem stays the same so it might be this reason. But still I think you were correct that there were some issues in our HPC, we will try to find and let you know once we have discovered the reason.

Thanks, 

Jiacheng