ImportError issue

[Arielle]

Hi all,

I’m not familiar with using python or python packages, so my problem
here may be something really trivial, but I am getting this error
after running JointSNVMix (this is in the very beginning of the
script):

/apps/Modules/3.2.8/init/bash_completion: line 14: syntax error near
unexpected token `('
/apps/Modules/3.2.8/init/bash_completion: line 14: ` comm -23
<(_module_avail|sort) <(tr : '\n' <<<${LOADEDMODULES}|sort)'
Traceback (most recent call last):
File "/srv/gs1/projects/li/shared-data/software/JointSNVMix-0.7.5/
jsm.py", line 9, in <module>
from joint_snv_mix.runners.train import snv_mix_one_train,
snv_mix_two_train, joint_snv_mix_one_train, \
File "/srv/gs1/projects/li/shared-data/software/JointSNVMix-0.7.5/
joint_snv_mix/runners/train.py", line 8, in <module>
from joint_snv_mix.counters.joint_binary_counter import
JointBinaryBaseCounter
ImportError: No module named joint_binary_counter

My PYTHONPATH environment variable is set to the joint_snv_mix python
modules. In the joint_snv_mix/counters directory, the
joint_binary_counter file has an .so suffix, rather than .py. I
thought that might be why python can’t identify that module, since I
thought python modules always end in .py, but I could be wrong about
this.

I would appreciate any help for this problem.

Thank you,
Arielle

[aroth]

Hi Arielle,
The .so suffix indicates that it is binary file, however Python can
support this. One issue maybe that you are using Python3, which I have
never tested on.

A little background. Python is undergoing a major overhaul of the
language so there are currently to versions Python2 and Python3. Both
versions are current and supported. JointSNVMix is tested on the
latest version from the Python2 branch, which is 2.7.2.

Here is how I install and run JointSNVMix on my labs cluster, where I
do NOT have administrative rights.

1) Create a folder where you want your installation to reside, say /
some/where/python_install
2) Create a sub-folder called src, so /some/where/python_install/src
3) Enter this folder and download the latest copy of python "wget
http://python.org/ftp/python/2.7.2/Python-2.7.2.tgz" will do this.
4) Uncompress the file Python-2.7.2.tgz, "tar -zxvf Python-2.7.2.tgz"
will do this.
5) Enter the directory created. "cd Python-2.7.2"
6) Python is a standard C package so it will be like a normal
compilation. One thing is we will specify to install into /some/where/
python_install.
a) Run "./configure --prefix=/some/where/python_install"
b) Run "make"
c) Run "make install"
7) There will now be a folder "/some/where/python_install/bin" with
the Python executable "python" in it.
To get this version of Python to be used in place of others you
need to add /some/where/python_install/bin to your PATH variable.
"export PATH=/some/where/python_install/bin:$PATH" will do this.
8) You can now install JointSNVMix as described and it will go to this
local install.

A couple points.

1) Step 7 is temporary, so you either a) have to add "export PATH=/
some/where/python_install/bin:$PATH" to your .bashrc/.profile or b)
execute that command each time you login.
2) If you are using a cluster this will only affect the node you
perform the commands on, usually the "Head Node". JointSNMVMix will
then only be available there.
To make this work across all nodes, "/some/where/python_install"
should be a shared folder visible across all nodes. Every clusters has
such a folder, you may need to ask your administrator what it is.

This installation method has the huge advantage that it does not touch
your cluster installs, or affect other users.

Cheers,
Andy

[Arielle]

Hi Andy,

Thank you for your reply. I am using my cluster's Python version 2.7,
so I'm not sure if the Python version is an issue here.
I don't know if this also helps, but when I just run Python 2.7 on the
command line and enter "import joint_binary_counter", I get this:

Traceback (most recent call last):

File "<stdin>", line 1, in <module>
ImportError: /path/to/python/modules/lib64/python/joint_snv_mix/
counters/joint_binary_counter.so: undefined symbol:
PyUnicodeUCS4_DecodeUTF8

Any other ideas here on what could be going on? If not then I will
try installing my own Python and see if that helps.

Thanks,
Arielle

On Feb 17, 7:17 pm, aroth <andrewjlr...@gmail.com> wrote:
> Hi Arielle,
> The .so suffix indicates that it is binary file, however Python can
> support this. One issue maybe that you are using Python3, which I have
> never tested on.
>
> A little background. Python is undergoing a major overhaul of the
> language so there are currently to versions Python2 and Python3. Both
> versions are current and supported. JointSNVMix is tested on the
> latest version from the Python2 branch, which is 2.7.2.
>
> Here is how I install and run JointSNVMix on my labs cluster, where I
> do NOT have administrative rights.
>
> 1) Create a folder where you want your installation to reside, say /
> some/where/python_install
> 2) Create a sub-folder called src, so /some/where/python_install/src

> 3) Enter this folder and download the latest copy of python "wgethttp://python.org/ftp/python/2.7.2/Python-2.7.2.tgz" will do this.

[aroth]

Hi Arielle,
Sorry about the delayed response. I haven't seen this problem so it
has taken a while to get an idea.

I have been searching around and I am not sure on what the issue is.
Searching for the error suggests that the Python interpreter used to
build the module is different than the one running it. This seems
unlikely since it appears to have been installed in the correct site-
packages folder.

I have one idea. When you installed JointSNVMix originally you used
and older version of Python. Did you delete the old build files before
running 'python setup.py install' with the new version of Python? If
not it might be worth deleting the unpacked JointSNVMix folder, and
decompressing it again to create a fresh copy, then run 'python
setup.py install'.

Cheers,
Andy

[Arielle]

Hi Andy,

Thanks again for your reply. I ended up re-installing everything
(python, cython, pysam, jointsnvmix) in my own directory, and updated
my PYTHONPATH variable. Now when I run "import joint_binary_counter"
on the python command line, it works fine. But after running the
jsm.py script, I still get the error:

" File "(localdirectory)/JointSNVMix-0.7.5/joint_snv_mix/runners/

train.py", line 8, in <module>
from joint_snv_mix.counters.joint_binary_counter import
JointBinaryBaseCounter
ImportError: No module named joint_binary_counter

I'm searching online to see if I can find some sort of solution, but
if you have any suggestions please let me know.

Thanks,
Arielle

[aroth]

Have you also set the PATH=(local_directory)/bin:$PATH? I have never
set the PYTHONPATH, I always just set PATH.

Hope that helps.

Cheers,
Andy

[Arielle]

Yes, I set that as well. Still trying to figure out the problem...

[Arielle]

I was originally just running the jsm.py that is located in the
JointSNVMix directory. However, I noticed that Python also put the
script in my python_install/bin directory. So I ran that one instead
and now I get a different error (the first two lines in this error
also occur when I run the jsm.py script in the JointSNVMix directory):

/srv/gs1/apps/Modules/3.2.8/init/bash_completion: line 14: syntax

error near unexpected token `('

/srv/gs1/apps/Modules/3.2.8/init/bash_completion: line 14: ` comm -23

<(_module_avail|sort) <(tr : '\n' <<<${LOADEDMODULES}|sort)'

Randomly sub-sampling every 100th position with normal depth 10 and
tumour depth 10 the data set.
Subsampling ref chrY.

Traceback (most recent call last):

File ".../python_install/bin/jsm.py", line 224, in <module>
args.func(args)
File ".../python_install/lib/python2.7/site-packages/joint_snv_mix/
runners/train.py", line 59, in joint_snv_mix_two_train
sample = ss.subsample(counter)
File "joint_snv_mix.pyx", line 40, in
joint_snv_mix.trainers.joint_snv_mix.PairedDataSubSampler.subsample
(joint_snv_mix/trainers/joint_snv_mix.c:3038)
File "joint_binary_quality_counter.pyx", line 27, in
joint_snv_mix.counters.joint_binary_quality_counter.JointBinaryQualityCounter.iter_ref
(joint_snv_mix/counters/joint_binary_quality_counter.c:1497)
File "quality_counter.pyx", line 24, in
joint_snv_mix.counters.quality_counter.QualityCounter.iter_ref
(joint_snv_mix/counters/quality_counter.c:1256)
File "csamtools.pyx", line 927, in csamtools.Samfile.pileup (pysam/
csamtools.c:9857)
ValueError: no index available for pileup

Is this a "better" error to have? Any suggestions?

Thanks,
Arielle

[aroth]

Thats a much better error, it means there is no index for either the
BAM or Fasta file. To index the BAM files just run 'samtools index
BAM_FILE' and for the Fasta run 'samtools faidx FASTA_FILE'.

I hope that fixes it.

Cheers,
Andy

[Arielle]

Looks like it all works now, thank you for your help! I did have the
index files, but they ended in .bai rather than .bam.bai. Once I
changed the suffix to .bam.bai, everything worked, although it still
only works when I run jsm.py script that is in the Python directory,
rather than the jointsnvmix directory.

Thanks again!

Arielle

> ...
>
> read more »

[aroth]

Hi Arielle,
I'm happy to hear that it works. The fact that jsm.py does not run
from the jointsnvmix directory is not surprising. The installer is
supposed to put a copy of jsm.py in your python_install/bin. The
reason is that if python_install/bin is set on the PATH variable,
typing jsm.py at the prompt will stat JointSNVMix.

This mean is you should just be able to call jsm.py from anywhere, you
do not have to change to the python_install/bin directory and call
`python jsm.py`.

Running `python jsm.py` inside the jointsnvmix folder is not how it is
supposed to be run. Maybe I should clarify that on the wiki.

Cheers,
Andy

> ...
>
> read more »