Error running S-Multixcan

[Qianhui Xu]

Hi experts,

Recently, while using MetaXcan, I encountered an issue when attempting to run SMultiXcan after successfully completing PrediXcan. The error is related to the SNP covariance file. To eliminate the possibility of a corrupted file, I repeatedly downloaded the file gtex_v8_expression_elastic_net_snp_smultixcan_covariance.txt.gz from https://zenodo.org/records/3519321, but the issue persists.

Below is the code I used:

python3 /home/mri/MetaXcan/software/SMulTiXcan.py \
--models_folder /home/mri/TWAS/elastic_net_eqtl/elastic_net_models \
--models_name_filter "en_(.*)db" \
--models_name_pattern "en_Brain_(.*).db" \
--snp_covariance /home/mri/TWAS/gtex_v8_expression_elastic_net_snp_smultixcan_covariance.txt.gz \
--metaxcan_folder /home/mri/TWAS/Spredict \
--metaxcan_filter "predixcan(.*).csv" \
--metaxcan_file_name_parse_pattern "predixcan_p1_(.*).csv" \
--gwas_file /home/mri/TWAS/p1_excluded_8_hg38_done.txt \
--snp_column panel_variant_id \
--effect_allele_column effect_allele \
--non_effect_allele_column non_effect_allele \
--pvalue_column pvalue \
--beta_column effect_size \
--keep_non_rsid \
--model_db_snp_key varID \
--cutoff_condition_number 30 \
--verbosity 1 \
--throw \
--output /home/mri/TWAS/multixcan/SMultiXcan_p1.csv

  Here are the output and error messages:

INFO - Creating context

INFO - Creating MetaXcan results manager
Level 9 - Loading metaxcan /home/mri/TWAS/Spredict/predixcan_p1_Amygdala.csv
Level 9 - Loading metaxcan /home/mri/TWAS/Spredict/predixcan_p1_Anterior_cingulate_cortex_BA24.csv
Level 9 - Loading metaxcan /home/mri/TWAS/Spredict/predixcan_p1_Caudate_basal_ganglia.csv
Level 9 - Loading metaxcan /home/mri/TWAS/Spredict/predixcan_p1_Cerebellar_Hemisphere.csv
Level 9 - Loading metaxcan /home/mri/TWAS/Spredict/predixcan_p1_Cerebellum.csv
Level 9 - Loading metaxcan /home/mri/TWAS/Spredict/predixcan_p1_Cortex.csv
Level 9 - Loading metaxcan /home/mri/TWAS/Spredict/predixcan_p1_Frontal_Cortex_BA9.csv
Level 9 - Loading metaxcan /home/mri/TWAS/Spredict/predixcan_p1_Hippocampus.csv
Level 9 - Loading metaxcan /home/mri/TWAS/Spredict/predixcan_p1_Hypothalamus.csv
Level 9 - Loading metaxcan /home/mri/TWAS/Spredict/predixcan_p1_Nucleus_accumbens_basal_ganglia.csv
Level 9 - Loading metaxcan /home/mri/TWAS/Spredict/predixcan_p1_Putamen_basal_ganglia.csv
Level 9 - Loading metaxcan /home/mri/TWAS/Spredict/predixcan_p1_Spinal_cord_cervical_c-1.csv
Level 9 - Loading metaxcan /home/mri/TWAS/Spredict/predixcan_p1_Substantia_nigra.csv
Level 9 - Building data
Level 9 - Processing Amygdala
Level 9 - Processing Anterior_cingulate_cortex_BA24
Level 9 - Processing Caudate_basal_ganglia
Level 9 - Processing Cerebellar_Hemisphere
Level 9 - Processing Cerebellum
Level 9 - Processing Cortex
Level 9 - Processing Frontal_Cortex_BA9
Level 9 - Processing Hippocampus
Level 9 - Processing Hypothalamus
Level 9 - Processing Nucleus_accumbens_basal_ganglia
Level 9 - Processing Putamen_basal_ganglia
Level 9 - Processing Spinal_cord_cervical_c-1
Level 9 - Processing Substantia_nigra
INFO - Loading genes
INFO - Context for snp covariance
INFO - Assessing GWAS-Models SNP intersection
INFO - Processing GWAS command line parameters
INFO - Reading input gwas: /home/mri/TWAS/p1_excluded_8_hg38_done.txt
/home/mri/MetaXcan/software/metax/gwas/Utilities.py:111: FutureWarning: In a future version of pandas all arguments of read_table except for the argument 'filepath_or_buffer' will be keyword-only
  keep_non_rsid=args.keep_non_rsid)
INFO - Processing input gwas
Level 9 - Using declared zscore
Level 9 - loading /home/mri/TWAS/elastic_net_eqtl/elastic_net_models/en_Brain_Amygdala.db
Level 9 - loading /home/mri/TWAS/elastic_net_eqtl/elastic_net_models/en_Brain_Anterior_cingulate_cortex_BA24.db
Level 9 - loading /home/mri/TWAS/elastic_net_eqtl/elastic_net_models/en_Brain_Caudate_basal_ganglia.db
Level 9 - loading /home/mri/TWAS/elastic_net_eqtl/elastic_net_models/en_Brain_Cerebellar_Hemisphere.db
Level 9 - loading /home/mri/TWAS/elastic_net_eqtl/elastic_net_models/en_Brain_Cerebellum.db
Level 9 - loading /home/mri/TWAS/elastic_net_eqtl/elastic_net_models/en_Brain_Cortex.db
Level 9 - loading /home/mri/TWAS/elastic_net_eqtl/elastic_net_models/en_Brain_Frontal_Cortex_BA9.db
Level 9 - loading /home/mri/TWAS/elastic_net_eqtl/elastic_net_models/en_Brain_Hippocampus.db
Level 9 - loading /home/mri/TWAS/elastic_net_eqtl/elastic_net_models/en_Brain_Hypothalamus.db
Level 9 - loading /home/mri/TWAS/elastic_net_eqtl/elastic_net_models/en_Brain_Nucleus_accumbens_basal_ganglia.db
Level 9 - loading /home/mri/TWAS/elastic_net_eqtl/elastic_net_models/en_Brain_Putamen_basal_ganglia.db
Level 9 - loading /home/mri/TWAS/elastic_net_eqtl/elastic_net_models/en_Brain_Spinal_cord_cervical_c-1.db
Level 9 - loading /home/mri/TWAS/elastic_net_eqtl/elastic_net_models/en_Brain_Substantia_nigra.db
INFO - Loading Model Manager
Level 9 - preloading models
Level 9 - processing Amygdala
Level 9 - processing Anterior_cingulate_cortex_BA24
Level 9 - processing Caudate_basal_ganglia
Level 9 - processing Cerebellar_Hemisphere
Level 9 - processing Cerebellum
Level 9 - processing Cortex
Level 9 - processing Frontal_Cortex_BA9
Level 9 - processing Hippocampus
Level 9 - processing Hypothalamus
Level 9 - processing Nucleus_accumbens_basal_ganglia
Level 9 - processing Putamen_basal_ganglia
Level 9 - processing Spinal_cord_cervical_c-1
Level 9 - processing Substantia_nigra
Level 9 - preparing models (dictionary layout)
INFO - Preparing SNP covariance
INFO - Processing
Traceback (most recent call last):
  File "/home/mri/MetaXcan/software/SMulTiXcan.py", line 93, in <module>
    run(args)
  File "/home/mri/MetaXcan/software/SMulTiXcan.py", line 31, in run
    for i,gene in enumerate(context.get_genes()):
  File "/home/mri/MetaXcan/software/metax/cross_model/Utilities.py", line 65, in get_genes
    for d in self.snp_covariance_streamer:
  File "/home/mri/MetaXcan/software/metax/misc/DataFrameStreamer.py", line 16, in data_frame_streamer
    header = file.readline().strip().split()
  File "/opt/anaconda3/envs/MetaXcan/lib/python3.7/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

  The covariance file seems to be a binary file. I tried converting it to UTF-8 encoding, but it still cannot be read correctly. Could you please provide me with some suggestions? Thank you!  

[Hae Kyung Im]

cc'ing Sofia who may be able to help

--
You received this message because you are subscribed to the Google Groups "PrediXcan/MetaXcan" group.
To unsubscribe from this group and stop receiving emails from it, send an email to predixcanmetax...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/predixcanmetaxcan/57fadebd-9175-4de6-8c13-704e3850060cn%40googlegroups.com.

[Sofía Salazar]

Hi,

It could be an issue with the python version. Could you please share which version you are using? I tested the problematic lines on versions 3.9.21 and 3.13.1 and I got the expected output. Could you try running this code as a test in your environment and a new environment with one of these python versions?

import io

import gzip

def _ogz(p):

return io.TextIOWrapper(gzip.open(p, "r"), newline="")

path="gtex_v8_expression_elastic_net_snp_smultixcan_covariance.txt.gz"

_o = _ogz if ".gz" in path else open

with _o(path) as file:

header = file.readline().strip().split()

print(header)

# expected output: ['GENE', 'RSID1', 'RSID2', 'VALUE']

[Qianhui Xu]

  Thank you for your prompt response. I tested the code Sogus provided on Python 3.7.16, Python 3.8.10, and Python 3.9.21, but all resulted in the same error. I don’t think the issue is related to the Python version, as the YAML file provided by the MetaXcan developers explicitly specifies Python 3.7, and I built my virtual environment based on that file. Thank you again for your suggestion, but I would appreciate further assistance.  

[Sofía Salazar]

Hi,
I understand. I am still unable to replicate this error, I created a new environment using the file provided here and tested the short code from before and it works for me. Would you please share your environment so I can try and replicate this error as faithfully as possible? 

conda env export --file environment.yaml

Best,

Sofia

[Qianhui Xu]

Hi，this is the output file. Thank you.

[Sofía Salazar]

Hi,

I have built and environment using the file you shared but I was still unable to replicate your error. Given this, I'm afraid I think the issue remains in the covariance file itself. I know you said you have re-downloaded the file repeatedly and the issue remains, so I would suggest verifying that you download process is not disrupted in any way as so to corrupt the covariance file. Also, perhaps trying running this using a different machine might be a solution. 

Please let me know if there's anything else I could do on our part to help,

Sofia