error with biom-format causing qiime to not work

[Daniel]

I was updating my biom-format package to the latest version (sudo pip install -upgrade biom-format)  and in doing so it seems to have disrupted my qiime install.

Now, if I enter the qiime shell and run the following:

sbi6dap@hive:~$  qiime > print_qiime_config.py -t
Traceback (most recent call last):
  File "/usr/local/bin/print_qiime_config.py", line 15, in <module>
    from qiime.util import make_option
  File "/usr/lib/python2.7/dist-packages/qiime/util.py", line 42, in <module>
    from biom.table import DenseTable
ImportError: cannot import name DenseTable

I haven't been able to get biom convert to wor since doing the upgrade either and I think something might have gone wrong.

Any ideas would be much appreciated.

Thanks

[Emily TerAvest]

Hi Daniel, 

The latest version of biom-format had a significant amount of API changes and will not work with the current release of qiime.  I recommend reverting back to the older version of biom-format for your qiime install. 

If you are interested in having both qiime and the latest version of biom-format you can use virtualenv to set up unique and separate python environments. 

Sorry about that 

Emily 

--

---
You received this message because you are subscribed to the Google Groups "Qiime Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qiime-forum...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel McDonald]

To add a little more, we will be getting BIOM 2.0 support in QIIME-dev in short order, but as Emily indicated, there are quite a few API changes which increases the complexity of the upgrade within QIIME.

Best,

Daniel 

[Daniel]

Thanks Emily & Daniel

I seem to be having a bit of difficulty removing the old biom files. What is the best way to do this?

I ran "sudo pip uninstall biom-format" but biom still pointed to /usr/local/bin/biom so I removed that directory.

I downloaded biom1.3.1 from https://pypi.python.org/packages/source/b/biom-format/biom-format-1.3.1.tar.gz as it says at http://qiime.org/install/install.html and ran "sudo python setup.py build" and "sudo python setup.py install"

but if I run biom convert --version I still get:

Version: biom convert 2.0.0

Also, if I try a simple convert:

Converting from classic to BIOM format: Convert the classic file table.txt to a sparse BIOM format OTU table
 biom convert -i table.txt -o table.biom --table-type "otu table"

I get the error

biom convert: error: no such option: --table-type

I imagine that new versions are persisting somewhere when I attempt to remove them. Any idea how to fix this?

Thanks

Daniel

[Daniel McDonald]

Hi Daniel,

Can you send the output from the following please?

python -c "import biom; print biom.__file__"

biom show-install-info

pip freeze

Thanks,

Daniel

--

[Daniel]

Thanks

From the look of it I'm swimming in 2.0.0 versions...

 

sbi6dap@hive:~/working$ python -c "import biom; print biom.__file__"

/usr/local/lib/python2.7/dist-packages/biom/__init__.pyc

sbi6dap@hive:~/working$ biom show-install-info

System information

==================

          Platform:     linux2

Python/GCC version:     2.7.3 (default, Feb 27 2014, 19:58:35)  [GCC 4.6.3]

 Python executable:     /usr/bin/python

Dependency versions

===================

 pyqi version:  0.3.2

NumPy version:  1.8.1

SciPy version:  0.14.0

 h5py version:  2.2.0

biom-format package information

===============================

biom-format version:    2.0.0

sbi6dap@hive:~/working$ pip freeze

Brlapi==0.5.6

CouchDB==0.8

Cython==0.19.1

GnuPGInterface==0.3.2

HTSeq==0.5.4p5

MACS==1.4.1

Mako==0.5.0

MarkupSafe==0.15

MySQL-python==1.2.3

PAM==0.4.2

PICRUSt==1.0.0-dev

PIL==1.1.7

PrimerProspector==1.0.1-release

PyICU==1.3

Pyste==0.9.10

Twisted-Core==11.1.0

Twisted-Names==11.1.0

Twisted-Web==11.1.0

adium-theme-ubuntu==0.3.2

apparmor==2.7.102

apt-clone==0.2.1

apt-xapian-index==0.44

apturl==0.5.1ubuntu3

argparse==1.2.1

biom-format==1.1.1-dev

chardet==2.0.1

cogent==1.5.3

command-not-found==0.2.44

configglue==1.0

debtagshw==0.1

defer==1.0.2

dirspec==3.0.0

duplicity==0.6.18

emperor==0.9.3

gnuplot-py==1.8

gufw==12.04.1

h5py==2.2.0

httplib2==0.7.2

jockey==0.9.7

keyring==0.9.2

khmer==0.7.1

language-selector==0.1

launchpadlib==1.9.12

lazr.restfulclient==0.12.0

lazr.uri==1.0.3

louis==2.3.0

matplotlib==1.1.1rc

mercurial==2.0.2

mpi4py==1.2.2

networkx==1.8.1

ngsutils==0.5.1a

numexpr==1.4.2

numpy==1.8.1

nvidia-common==0.0.0

oauth==1.0.1

onboard==0.97.1

oneconf==0.2.8.1

pbcore==0.8.0

pbtools.pbdagcon==0.2.3

pbtools.pbh5tools==0.75.0

pexpect==2.3

piston-mini-client==0.7.2

protobuf==2.4.1

pyOpenSSL==0.12

pycrypto==2.4.1

pycups==1.9.61

pycurl==7.19.0

pyinotify==0.9.2

pynast==1.2.2

pyparsing==1.5.2

pyqi==0.3.2

pysam==0.7.5

pyserial==2.5

pysmbc==1.0.13

python-apt==0.8.3ubuntu7.1

python-dateutil==1.5

python-debian==0.1.21ubuntu1

python-virtkey==0.60.0

pytz==2011k

pyxdg==0.19

qcli==0.1.0

rdflib==2.4.2

reportlab==2.5

rhythmbox-ubuntuone==3.0.0

rpy==1.0.3

rpy2==2.3.6

scipy==0.14.0

screed==0.7.1

screen-resolution-extra==0.0.0

sessioninstaller==0.0.0

simplejson==2.3.2

software-center-aptd-plugins==0.0.0

system-service==0.1.6

tables==2.3.1

tax2tree==1.0

ubuntuone-couch==0.3.0

ubuntuone-installer==3.0.2

ubuntuone-storage-protocol==3.0.2

ufw==0.31.1-1

unattended-upgrades==0.1

unity-lens-video==0.3.5

unity-scope-video-remote==0.3.5

unity-singlet==0.2.2

usb-creator==0.2.23

virtualenv==1.7.1.2

wadllib==1.3.0

weblogo==3.1

wsgiref==0.1.2

xdiagnose==2.5.3

xkit==0.0.0

zope.interface==3.6.1

[Daniel McDonald]

That's quite a bit in pip, as a general comment, I do recommend at some point in the future getting familiar with virtualenv and virtualenvwrapper as they are absolutely fantastic for managing environments in Python.

But, let's resolve the current issue. Can you please send the output from the following:

ls -d /usr/local/lib/python2.7/dist-packages/biom*

Thanks,

Daniel

--

[Daniel]

Thanks for the advice. I don't know what 75% of them are although I can see some that I put in there. This is a shared bio-linux cluster so I imagine some got introduced by others.

here's the output:

sbi6dap@hive:~/working$ ls -d /usr/local/lib/python2.7/dist-packages/biom*

/usr/local/lib/python2.7/dist-packages/biom                            

/usr/local/lib/python2.7/dist-packages/biom_format-1.3.1.egg-info

/usr/local/lib/python2.7/dist-packages/biom_format-1.1.1_dev.egg-info 

/usr/local/lib/python2.7/dist-packages/biom_format-2.0.0-py2.7.egg-info

[Daniel McDonald]

I really recommend maintaining your own environment then under $HOME, or if there is a sysadmin, requesting that they establish modules or dotkit for environment management. Because this is a shared cluster, and the potential for changes to impact other users of the system, I'm hesitant to recommend any changes at the system level. As a practical concern, any other user of the system can modify your environment without your knowledge, which can lead to interesting problems. Let's do a local install of BIOM instead. 

1) change back to the unpacked biom-format-1.3.1 directory

2) run "python setup.py clean"

3) run "python setup.py build"

4) run "python setup.py install --prefix=$HOME"

'sudo' is not be necessary for any of this, with the exception possibly of step 2 as, if you did the build previously using 'sudo' then the files created under the build/ directory are owned by the root user. These above steps will install BIOM under $HOME. We aren't quite done yet though, once this is done, can you please send the output from the following commands?

which biom

echo $PATH

echo $PYTHONPATH

Best,

Daniel

 

[Daniel]

OK, so I looked at my PATH and had a little heart attack. It turns out I had a line in there to source a qiime/activate.sh script from when I ran my own local qiime version. I've removed it below.

Generally I'm the one that will run updates on our cluster and others will use them, or I'll be asked to update them at some later point which is why I went straight for global install. In hindsight, I shouldn't have done it until I knew it worked but hey, that's in the past now!

So I get a couple of warnings on build:

python setup.py build

/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'extras_require'

  warnings.warn(msg)

/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'install_requires'

  warnings.warn(msg)

python setup.py  install --prefix=$HOME
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'extras_require'
  warnings.warn(msg)
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'install_requires'
  warnings.warn(msg)

......................... [normal compiling continues].................................

 

Original Paths:

sbi6dap@hive:~/programs/biom-format1.3/biom-format-1.3.1$ which biom

/usr/local/bin/biom

sbi6dap@hive:~/programs/biom-format1.3/biom-format-1.3.1$ echo $PATH

/home/sbi6dap/working/qiime_software/r-3.0.2-release/bin:/home/sbi6dap/working/qiime_software/cytoscape-2.7.0-release/.:/home/sbi6dap/working/qiime_software/cdhit-3.1-release/.:/home/sbi6dap/working/qiime_software/rdpclassifier-2.2-release/.:/home/sbi6dap/working/qiime_software/SeqPrep-latest-repository-85e2389e/.:/home/sbi6dap/working/qiime_software/blast-2.2.22-release/bin:/home/sbi6dap/working/qiime_software/muscle-3.8.31-release/.:/home/sbi6dap/working/qiime_software/vienna-1.8.4-release/.:/home/sbi6dap/working/qiime_software/parsinsert-1.0.4-release/.:/home/sbi6dap/working/qiime_software/infernal-1.0.2-release/bin:/home/sbi6dap/working/qiime_software/bwa-0.6.2-release/.:/home/sbi6dap/working/qiime_software/chimeraslayer-4.29.2010-release/ChimeraSlayer:/home/sbi6dap/working/qiime_software/chimeraslayer-4.29.2010-release/NAST-iEr:/home/sbi6dap/working/qiime_software/qiime-galaxy-0.0.1-repository-c2814c3c/scripts:/home/sbi6dap/working/qiime_software/mothur-1.25.0-release/.:/home/sbi6dap/working/qiime_software/ea-utils-1.1.2-537-release/.:/home/sbi6dap/working/qiime_software/drisee-1.2-release/.:/home/sbi6dap/working/qiime_software/raxml-7.3.0-release/.:/home/sbi6dap/working/qiime_software/ampliconnoise-1.27-release/Scripts:/home/sbi6dap/working/qiime_software/ampliconnoise-1.27-release/bin:/home/sbi6dap/working/qiime_software/pprospector-1.0.1-release/bin:/home/sbi6dap/working/qiime_software/rtax-0.984-release/.:/home/sbi6dap/working/qiime_software/sourcetracker-0.9.5-release/.:/home/sbi6dap/working/qiime_software/cdbtools-10.11.2010-release/.:/home/sbi6dap/working/qiime_software/tax2tree-1.0-release/bin:/home/sbi6dap/working/qiime_software/clearcut-1.0.9-release/.:/home/sbi6dap/working/qiime_software/blat-34-release/.:/home/sbi6dap/picrust-dev/scripts/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/bioinf/mothur/mothur/bin:/usr/local/bioinf/bowtie/bowtie:/usr/local/bioinf/tophat/tophat:/usr/local/bioinf/cufflinks/cufflinks:/home/projects/smbpk/shared_scripts:/home/projects/smbpk/shared_scripts/tims/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/bioinf/mothur/mothur/bin:/usr/local/bioinf/bowtie/bowtie:/usr/local/bioinf/tophat/tophat:/usr/local/bioinf/bowtie/bowtie2:/usr/local/bioinf/cufflinks/cufflinks:/usr/software/oases/oases:/usr/software/git/git:/usr/software/velvet/velvet:/usr/software/picrust/picrust/scripts:/usr/software/minia/minia:/usr/local/bioinf/samtools/samtools/misc:/usr/local/bin:/usr/software/transabyss/transabyss/bin:/java/path:/usr/software/abyss/abyss:/usr/software/transabyss/transabyss::/usr/software/picrust/picrust/picrust:/usr/software/ngutils/ngsutils/bin:/usr/software/rnnotator/rnnotator/scripts

sbi6dap@hive:~/programs/biom-format1.3/biom-format-1.3.1$ echo $PYTHONPATH

/home/sbi6dap/working/qiime_software/qiime-galaxy-0.0.1-repository-c2814c3c/lib/:/home/sbi6dap/working/qiime_software/tax2tree-1.0-release/lib/python2.7/site-packages:/home/sbi6dap/working/qiime_software/pprospector-1.0.1-release/lib/python2.7/site-packages:/home/sbi6dap/picrust-dev/picrust/:/usr/lib/pyshared/:/usr/software/transabyss/transabyss::/usr/software/picrust/picrust/picrust

After removing the qiime/activate.sh from my .bashrc:

sbi6dap@hive:~$ echo $PATH

/home/sbi6dap/bin:/home/sbi6dap/picrust-dev/scripts/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/bioinf/mothur/mothur/bin:/usr/local/bioinf/bowtie/bowtie:/usr/local/bioinf/tophat/tophat:/usr/local/bioinf/cufflinks/cufflinks:/home/projects/smbpk/shared_scripts:/home/projects/smbpk/shared_scripts/tims/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/bioinf/mothur/mothur/bin:/usr/local/bioinf/bowtie/bowtie:/usr/local/bioinf/tophat/tophat:/usr/local/bioinf/bowtie/bowtie2:/usr/local/bioinf/cufflinks/cufflinks:/usr/software/oases/oases:/usr/software/git/git:/usr/software/velvet/velvet:/usr/software/picrust/picrust/scripts:/usr/software/minia/minia:/usr/local/bioinf/samtools/samtools/misc:/usr/local/bin:/usr/software/transabyss/transabyss/bin:/java/path:/usr/software/abyss/abyss:/usr/software/transabyss/transabyss::/usr/software/picrust/picrust/picrust:/usr/software/ngutils/ngsutils/bin:/usr/software/rnnotator/rnnotator/scripts

sbi6dap@hive:~$ echo $PYTHONPATH

/home/sbi6dap/picrust-dev/picrust/:/usr/lib/pyshared/:/usr/software/transabyss/transabyss::/usr/software/picrust/picrust/picrust

[Daniel McDonald]

Given your environment, I suspect you'll need to keep the activate.sh line in .bashrc. However, what we need to do is make sure that $PATH has $HOME/bin first, and that $PYTHONPATH has $HOME/lib/python2.7/site-packages first. Do you know how to make those modifications or would you like a step-by-step break down? (just asking since you were already toying with .bashrc)

-Daniel

[Daniel]

Thanks, I've added them to the $PATH and $PYTHONPATH respectively in .bashrc

"which biom" now points to the new files:

/home/sbi6dap/bin/biom

Although I still get the following:

sbi6dap@hive:~$ python -c "import biom; print biom.__file__"
/usr/local/lib/python2.7/dist-packages/biom/__init__.pyc

 

sbi6dap@hive:~$ biom show-install-info

[Sebastian Lau]

Hi Daniel, I have the same problem this post mentioned. I try to revert back to biom 1.3 following your suggestion and then I got this ：

SEBASTIANs-MacBook-Pro:~ FLFLFLLF$ which biom

/usr/local/bin/biom

SEBASTIANs-MacBook-Pro:~ FLFLFLLF$ echo $PATH

/Library/Frameworks/Python.framework/Versions/2.7/bin:/opt/local/bin:/opt/local/sbin:/opt/blast-2.2.22/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin

SEBASTIANs-MacBook-Pro:~ FLFLFLLF$ echo $PYTHONPATH

SEBASTIANs-MacBook-Pro:~ FLFLFLLF$ 

Then I checked the biom version again and got this:

SEBASTIANs-MacBook-Pro:~ FLFLFLLF$ python -c "import biom; print biom.__file__"

Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File "/Library/Python/2.7/site-packages/biom/__init__.py", line 24, in <module>

    from .table import Table

  File "/Library/Python/2.7/site-packages/biom/table.py", line 176, in <module>

    import numpy as np

ImportError: No module named numpy

SEBASTIANs-MacBook-Pro:~ FLFLFLLF$ biom show-install-info

pip freeze

/Library/Python/2.7/site-packages/biom/util.py:27: UserWarning: h5py is not available

  warn("h5py is not available")

System information

==================

          Platform: darwin

Python/GCC version: 2.7.5 (default, Mar  9 2014, 22:15:05)  [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)]

 Python executable: /usr/bin/python

Dependency versions

===================

 pyqi version: 0.3.2

NumPy version: 1.6.2

SciPy version: 0.11.0

 h5py version: WARNING: Not installed - this is an optional dependency. It is strongly recommended for large datasets.

biom-format package information

===============================

biom-format version: 2.0.0

seems I didn't get biom 1.3 installed.

what should I do now? Thanks 

[Daniel McDonald]

Hi Sebastian, your issue was different in that it appeared the input file you were trying to convert was not a format that BIOM understands, not an issue with the version of BIOM being used. Can you please send the commands you issued when trying to get a different version of BIOM installed?

-Daniel

--

[Daniel McDonald]

Interesting. It is possible the order is thrown off in your PYTHONPATH still (qiime's activate.sh may do this). Can you please verify $PYTHONPATH again?

-Daniel

[Daniel]

nope, it's first up:

sbi6dap@hive:~$ echo $PYTHONPATH
/home/sbi6dap/lib/python2.7/site-packages/:/picrust-dev/picrust/:/usr/lib/pyshared/:/usr/software/transabyss/transabyss::/usr/software/picrust/picrust/picrust

I really appreciate all the effort you're going to.

[Daniel]

Also, just to check, I shouldn't be worrying about this?:

[Daniel]

I'm still trying to fix this, but I think I might have found the problem. My local biom-format install is fine, but when it calls the pyqi it goes out to the system install which I think is calling the incorrect biom version from somewhere.

So the command from biom version 1.3 (in $HOME/bin/biom):

exec pyqi --driver-name biom --command-config-module biom.interfaces.optparse.config -- "$@"

Points to the system pyqi (in the script it says __version__ = "0.2.0-dev" but in the pip freeze it says 0.3.2)

I'm trying to find where pyqi pulls the biom modules from after passing the --driver-name as it must be looking somewhere other than $HOME/lib but I'm not sure how to do it.

As a side note, the biom-format site says that for version 2.0.0 I should have the following, but I can't find the requirements for the biom-format v1.3 which I'm trying to revert to. Would you think that any of these would break backwards compatibility?

Requirements for 2.0.0

Python >= 2.7 and < 3.0

numpy >= 1.7.0
pyqi 0.3.2
scipy >= 0.13.0
h5py >= 2.20.0 

What I actually have according to pip freeze:

Python == 2.7

numpy==1.8.1

pyqi==0.3.2

scipy==0.14.0

h5py==2.2.0

[Daniel]

I think I have solved the problem quite simply in the end, by retrieving the pip repository for version 1.3.0 and forcing an install over the newer versions like so:

sudo pip uninstall biom-format

sudo pip install -Iv https://pypi.python.org/packages/source/b/biom-format/biom-format-1.3.0.tar.gz#md5=bd35f9b7e1c7daf244fe2dffb2777606

Thanks for all the assistance.

[Daniel McDonald]

Hey Daniel,

Sorry for the delayed response, was tied up yesterday afternoon. It does appear there are conflicting versions of pyqi now, and I agree it could be related to where biom is sourced. Can you send on:

which pyqi

python -c "import pyqi; print pyqi.__file__"

Sorry for all the challenges here...

-Daniel

[Daniel McDonald]

Also, the specific BIOM v1.3.1 requirements can be see here, and the 1.3.1 documentation that mentions the requirements can be found here.

-Daniel

[Daniel McDonald]

The bruteforce way :) That approach makes my sysadmin-side twitch, but if you're comfortable with it then charge ahead.

Best,

Daniel

--