ERROR running split_libraries_fastq.py (TypeError: numpy boolean subtract, the `-` operator....)
649 views
Skip to first unread message
Yuanyuan
Jun 21, 2017, 7:34:00 AM6/21/17
to Qiime 1 Forum
Hi everyone,
I recently installed qiime 1.9.1 on my Linux ubuntu 16.04 system. Errors kept occurring when I try to run split_libraries_fastq.py. But nothing went wrong when I run this command previously using Virtual Box. Was there anything wrong or missed when I installed qiime on Linux?
The warning of running split_libraries_fastq.py is as follows:
(qiime191) tyy@tyy-ThinkPad-T420:~/QIIME_practice/moving_pictures_tutorial-1.9.0$ split_libraries_fastq.py -i forward_reads.fastq -b barcodes.fastq -m map.tsv_c
orrected.txt -o slout
Traceback (most recent call last):
File "/home/tyy/anaconda2/envs/qiime191/bin/split_libraries_fastq.py", line 4, in <module>
__import__('pkg_resources').run_script('qiime==1.9.1', 'split_libraries_fastq.py')
File "/home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 744, in run_script
File "/home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 1499, in run_script
File "/home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime-1.9.1-py2.7.egg-info/scripts/split_libraries_fastq.py", line 365, in <module>
main()
File "/home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime-1.9.1-py2.7.egg-info/scripts/split_libraries_fastq.py", line 344, in main
for fasta_header, sequence, quality, seq_id in seq_generator:
File "/home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime/split_libraries_fastq.py", line 373, in process_fastq_single_end_read_file
filter_bad_illumina_qual_digit)
File "/home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime/split_libraries_fastq.py", line 127, in quality_filter_sequence
phred_quality_threshold)
File "/home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime/split_libraries_fastq.py", line 93, in read_qual_score_filter
for starts, ends in _contiguous_regions(mask):
File "/home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime/split_libraries_fastq.py", line 68, in _contiguous_regions
d = np.diff(condition)
File "/home/tyy/.local/lib/python2.7/site-packages/numpy/lib/function_base.py", line 1926, in diff
return a[slice1]-a[slice2]
TypeError: numpy boolean subtract, the `-` operator, is deprecated, use the bitwise_xor, the `^` operator, or the logical_xor function instead.
Further more, the output of running print_qiime_config.py -tf is:
(qiime191) tyy@tyy-ThinkPad-T420:~/QIIME_practice/moving_pictures_tutorial-1.9.0$ print_qiime_config.py -tf
System information
==================
Platform: linux2
Python version: 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:09:15) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
Python executable: /home/tyy/anaconda2/envs/qiime191/bin/python
QIIME default reference information
===================================
For details on what files are used as QIIME's default references, see here:
Dependency versions
===================
QIIME library version: 1.9.1
QIIME script version: 1.9.1
qiime-default-reference version: 0.1.3
NumPy version: 1.13.0
SciPy version: 0.17.1
pandas version: 0.18.1
matplotlib version: 1.4.3
biom-format version: 2.1.5
h5py version: 2.6.0 (HDF5 version: 1.8.16)
qcli version: 0.1.1
pyqi version: 0.3.2
scikit-bio version: 0.2.3
PyNAST version: 1.2.2
Emperor version: 0.9.51
burrito version: 0.9.1
burrito-fillings version: 0.1.1
sortmerna version: SortMeRNA version 2.0, 29/11/2014
sumaclust version: SUMACLUST Version 1.0.00
swarm version: Swarm 2.1.6 [Dec 17 2015 07:36:40]
gdata: Installed.
RDP Classifier version (if installed): rdp_classifier-2.2.jar
Java version (if installed): Not installed.
QIIME config values
===================
For definitions of these settings and to learn how to configure QIIME, see here:
blastmat_dir: None
pick_otus_reference_seqs_fp: /home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
sc_queue: all.q
topiaryexplorer_project_dir: None
pynast_template_alignment_fp: /home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.pynast.fasta
cluster_jobs_fp: start_parallel_jobs.py
pynast_template_alignment_blastdb: None
assign_taxonomy_reference_seqs_fp: /home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
torque_queue: friendlyq
jobs_to_start: 1
slurm_time: None
denoiser_min_per_core: 50
assign_taxonomy_id_to_taxonomy_fp: /home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt
temp_dir: /tmp/
slurm_memory: None
slurm_queue: None
blastall_fp: blastall
seconds_to_sleep: 1
QIIME full install test results
===============================
...FF....FF.F....FF..FFF..F
======================================================================
FAIL: test_ampliconnoise_install (__main__.QIIMEDependencyFull)
AmpliconNoise install looks sane.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime-1.9.1-py2.7.egg-info/scripts/print_qiime_config.py", line 384, in test_ampliconnoise_install
"$PYRO_LOOKUP_FILE variable is not set to an existing filepath.")
AssertionError: $PYRO_LOOKUP_FILE variable is not set to an existing filepath.
======================================================================
FAIL: test_blast_supported_version (__main__.QIIMEDependencyFull)
blast is in path and version is supported
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime-1.9.1-py2.7.egg-info/scripts/print_qiime_config.py", line 470, in test_blast_supported_version
% ('.'.join(map(str, acceptable_version)), version_string))
AssertionError: Unsupported blast version. 2.2.22 is required, but running 2.2.26.
======================================================================
FAIL: test_chimeraSlayer_install (__main__.QIIMEDependencyFull)
no obvious problems with ChimeraSlayer install
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime-1.9.1-py2.7.egg-info/scripts/print_qiime_config.py", line 429, in test_chimeraSlayer_install
self.assertTrue(chim_slay, "ChimeraSlayer was not found in your $PATH")
AssertionError: ChimeraSlayer was not found in your $PATH
======================================================================
FAIL: test_clearcut_supported_version (__main__.QIIMEDependencyFull)
clearcut is in path and version is supported
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime-1.9.1-py2.7.egg-info/scripts/print_qiime_config.py", line 601, in test_clearcut_supported_version
"which components of QIIME you plan to use.")
AssertionError: clearcut not found. This may or may not be a problem depending on which components of QIIME you plan to use.
======================================================================
FAIL: test_denoiser_supported_version (__main__.QIIMEDependencyFull)
denoiser aligner is ready to use
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime-1.9.1-py2.7.egg-info/scripts/print_qiime_config.py", line 570, in test_denoiser_supported_version
"Denoiser flowgram aligner not found or not "
AssertionError: Denoiser flowgram aligner not found or not executable. This may or may not be a problem depending on which components of QIIME you plan to use.
======================================================================
FAIL: test_mothur_supported_version (__main__.QIIMEDependencyFull)
mothur is in path and version is supported
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime-1.9.1-py2.7.egg-info/scripts/print_qiime_config.py", line 558, in test_mothur_supported_version
% ('.'.join(map(str, acceptable_version)), version_string))
AssertionError: Unsupported mothur version. 1.25.0 is required, but running 1.38.1.
======================================================================
FAIL: test_muscle_supported_version (__main__.QIIMEDependencyFull)
muscle is in path and version is supported
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime-1.9.1-py2.7.egg-info/scripts/print_qiime_config.py", line 533, in test_muscle_supported_version
% ('.'.join(map(str, acceptable_version)), version_string))
AssertionError: Unsupported muscle version. 3.8.31 is required, but running 3.8.1551.
======================================================================
FAIL: test_raxmlHPC_supported_version (__main__.QIIMEDependencyFull)
raxmlHPC is in path and version is supported
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime-1.9.1-py2.7.egg-info/scripts/print_qiime_config.py", line 580, in test_raxmlHPC_supported_version
"which components of QIIME you plan to use.")
AssertionError: raxmlHPC not found. This may or may not be a problem depending on which components of QIIME you plan to use.
======================================================================
FAIL: test_rtax_supported_version (__main__.QIIMEDependencyFull)
rtax is in path and version is supported
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime-1.9.1-py2.7.egg-info/scripts/print_qiime_config.py", line 629, in test_rtax_supported_version
"which components of QIIME you plan to use.")
AssertionError: rtax not found. This may or may not be a problem depending on which components of QIIME you plan to use.
======================================================================
FAIL: test_sourcetracker_installed (__main__.QIIMEDependencyFull)
sourcetracker is installed
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime-1.9.1-py2.7.egg-info/scripts/print_qiime_config.py", line 412, in test_sourcetracker_installed
("SOURCETRACKER_PATH is not set. This is "
AssertionError: SOURCETRACKER_PATH is not set. This is only important if you plan to use SourceTracker.
======================================================================
FAIL: test_usearch_supported_version (__main__.QIIMEDependencyFull)
usearch is in path and version is supported
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tyy/anaconda2/envs/qiime191/lib/python2.7/site-packages/qiime-1.9.1-py2.7.egg-info/scripts/print_qiime_config.py", line 650, in test_usearch_supported_version
"which components of QIIME you plan to use.")
AssertionError: usearch not found. This may or may not be a problem depending on which components of QIIME you plan to use.
----------------------------------------------------------------------
Ran 27 tests in 0.093s
FAILED (failures=11)
How can I fix this?
Many thanks!
Yuanyuan
justink
Jun 21, 2017, 5:35:55 PM6/21/17
to Qiime 1 Forum
I'm not sure, but can I suggest that you install qiime with miniconda? http://qiime.org/install/install.html
justink
Jun 21, 2017, 5:38:53 PM6/21/17
to Qiime 1 Forum
I *suspect* it's a numpy version error: you *could* try downgrading numpy to 1.10.4. But I'd recommend the miniconda approach.
Yuanyuan
Jun 22, 2017, 2:38:29 AM6/22/17
to Qiime 1 Forum
Dear Justink,
Thank you for you suggestion. I reinstalled qiime with miniconda. It still returns the same error.
(qiime1) tyy@tyy-ThinkPad-T420:~/QIIME_practice/moving_pictures_tutorial-1.9.0$ split_libraries_fastq.py -i forward_reads.fastq -b barcodes.fastq -m map.tsv_corrected.txt -o slout
Traceback (most recent call last):
File "/home/tyy/miniconda2/envs/qiime1/bin/split_libraries_fastq.py", line 4, in <module>
__import__('pkg_resources').run_script('qiime==1.9.1', 'split_libraries_fastq.py')
File "/home/tyy/miniconda2/envs/qiime1/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 744, in run_script
File "/home/tyy/miniconda2/envs/qiime1/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 1499, in run_script
File "/home/tyy/miniconda2/envs/qiime1/lib/python2.7/site-packages/qiime-1.9.1-py2.7.egg-info/scripts/split_libraries_fastq.py", line 365, in <module>
main()
File "/home/tyy/miniconda2/envs/qiime1/lib/python2.7/site-packages/qiime-1.9.1-py2.7.egg-info/scripts/split_libraries_fastq.py", line 344, in main
for fasta_header, sequence, quality, seq_id in seq_generator:
File "/home/tyy/miniconda2/envs/qiime1/lib/python2.7/site-packages/qiime/split_libraries_fastq.py", line 373, in process_fastq_single_end_read_file
filter_bad_illumina_qual_digit)
File "/home/tyy/miniconda2/envs/qiime1/lib/python2.7/site-packages/qiime/split_libraries_fastq.py", line 127, in quality_filter_sequence
phred_quality_threshold)
File "/home/tyy/miniconda2/envs/qiime1/lib/python2.7/site-packages/qiime/split_libraries_fastq.py", line 93, in read_qual_score_filter
for starts, ends in _contiguous_regions(mask):
File "/home/tyy/miniconda2/envs/qiime1/lib/python2.7/site-packages/qiime/split_libraries_fastq.py", line 68, in _contiguous_regions
justink
Jun 22, 2017, 4:50:39 PM6/22/17
to Qiime 1 Forum
does seem like we should fix that dependency on boolean subtract.
I'd say run print_qiime_config.py again: numpy version should be 1.10.x .
Maybe try:
"conda create -n qiime2 numpy=1.10 python=2.7 qiime matplotlib=1.4.3 mock nose -c bioconda" to force it.
Sarah Strauss
Jun 23, 2017, 2:30:50 PM6/23/17
to Qiime 1 Forum
I also just ran into this problem. I'm running QIIME on a university cluster and don't easily have access to changing the numpy versions. I ran multiple_split_libraries on the same data set 2 days ago and didn't have any problems. Now I'm getting the same error as Yuanyuan. Thoughts?
Thanks,
Sarah
Yuanyuan
Jun 24, 2017, 8:56:23 AM6/24/17
to Qiime 1 Forum
I gave it up to install qiime on Ubuntu 16.04.
Instead of that, I installed Bio-linux 8, which was particularly developed for bioinformatics based on Ubuntu 14.04 version. Qiime and its dependencies can be easily installed from the Synaptic Package Manager in Bio-linux 8. I did some test and no errors showed again. But I am now just using my laptop to install this pipeline and do some practices.
I am still worried about errors if natively install qiime on our server future when I need to deal with some big datasets.
Any suggestions about install qiime on a server?
Many thanks,
Yuanyuan
Instead of that, I installed Bio-linux 8, which was particularly developed for bioinformatics based on Ubuntu 14.04 version. Qiime and its dependencies can be easily installed from the Synaptic Package Manager in Bio-linux 8. I did some test and no errors showed again. But I am now just using my laptop to install this pipeline and do some practices.
I am still worried about errors if natively install qiime on our server future when I need to deal with some big datasets.
Any suggestions about install qiime on a server?
Many thanks,
Yuanyuan
Colin Brislawn
Jun 24, 2017, 1:47:23 PM6/24/17
to qiime...@googlegroups.com
Hello Yuanyuan, Sarah,
I've had a really good experience installing qiime using the conda platform. This systems works great, it it works equally well on osx, linux, and linux supercomputing clusters (even when you don't have admin access!!).
Conda (short for 'anaconda') lets you install software into independant software environments. This is super important because it allows you to use the exact versions of software that you need, without worrying about what other people have installed on the computer.
The install itself is easy. The command starts like this:
conda create -n qiime1
This uses the conda program to create a new environments called 'qiime2'. Then you list the other pieces of software you want to install, like this:
conda create -n qiime1 numpy=1.10 python=2.7 qiime matplotlib=1.4.3 mock nose -c bioconda
Notice how you can list specific versions (numpy=1.10) or just the package name by itself (mock).
When you want to run qiime scripts, simply active the new conda environment.
source activate qiime1
So try using conda. It's a great way to install qiime on all your systems.
Colin
Message has been deleted
tao yang
Jun 26, 2017, 5:02:29 PM6/26/17
to Qiime 1 Forum
This happened to me, too! It is a univrersity based cluster center.
I run split_libraries_fastq and it was fine until last Friday. I saw the same error...
Thanks,
Tao
Colin Brislawn
Jun 26, 2017, 7:46:49 PM6/26/17
to Qiime 1 Forum
Hello Tao,
Sorry to hear that you are having this problem as well. Hopefully we can find a solution.
On this university cluster, are you running a qiime installation provided by your university, or are you running qiime through a conda environment?
If they set up qiime, you could have them fix this. I think the conda environment should still work for you, regardless what they are using. This may be a great time for you to try conda!
Colin
Chris Hale
Jun 27, 2017, 5:39:53 AM6/27/17
to Qiime 1 Forum
Hello all,
Just thought I'd add that I had the same error as YuanYuan: TypeError: numpy boolean subtract, the `-` operator, is deprecated, use the bitwise_xor, the `^` operator, or the logical_xor function instead.
We were running the normal Qiime install on a linux server.
We followed the conda install of qiime2, specifying numpy 1.10 and it is working now. ( As suggested by Justin and Colin)
Thanks for the advice,
Chris
Colin Brislawn
Jun 27, 2017, 10:59:00 AM6/27/17
to Qiime 1 Forum
Glad it's working for you Chris. Conda is THE way to install qiime these days.
I would like to make a suggestion about my install command:
conda create -n qiime1 numpy=1.10 python=2.7 qiime matplotlib=1.4.3 mock nose -c bioconda
source activate qiime1
(I just changed the environment name to qiime1, because there will be a qiime2, with different requirements!)
Colin
Yuanyuan
Jul 6, 2017, 1:11:56 AM7/6/17
to qiime...@googlegroups.com
Hi Justink and Colin,
Thank you very much for your reply about the error I proposed last time.
Our server is kind of out of memory... So, sadly, I am not allowed to install qiime on it.
Instead, I got a new PC whose system information is as follows,
Processor: Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz 3.60 GHz
Installed memory (RAM): 16.0 GB (15.9 GB usable)
System type: 64-bit Operating system, X64-based processor
Now I install qiime with VB on this PC and allocated 14GB of RAM to the virtual machine, but I got a new problem.
It always got killed when running split_libraries.py with a relative larger dataset. The ***.fna file is around 4GB and the ***.qual file is around 11 GB. But it is fine when I ignore the quality file, just run the command like split_libraries.py -m *** -f ***.fna -o ***. Is my PC not powered enough to analyze such large dataset?
I found some similar problems proposed by others previously.
As they said, the quality file can be loaded into memory to cause memory issues.
Any solutions about that?
Yuanyuan
justink
Jul 6, 2017, 1:40:27 AM7/6/17
to Qiime 1 Forum
Hmm, sounds like some poor coding on our part. I guess we weren't smart enough to read in the files a few lines at a time.
Anyway, 2 options I can think of:
1. rent a big beefy machine on amazon: http://qiime.org/tutorials/working_with_aws.html
2. manually split the files: easiest if you have the fastq file, otherwise make sure if you split the .fasta file, you split the .qual file at the same place. Commands like:
head -n 1000 seqs.fasta
grep -c '>' seqs.fasta
split seqs.fasta
might help.
Reply all
Reply to author
Forward
0 new messages