Issue with normalize_by_copy_number.py

[Austin Ly]

Hello,

I recently installed the new PICRUSt version 1.1.0 and added in the two precalculated files into the folder /picrust_1.1.0/picrust/data, as instructed. I tried running normalize_by_copy_number.py but I got an error, as shown below. I would greatly appreciate any help with this issue. Thank you very much in advance!

MacQIIME nutritions-MacBook-Air:picrust  $ normalize_by_copy_number.py -i otus.biom -o normalized_otus.biom

Traceback (most recent call last):

  File "/Library/Frameworks/Python.framework/Versions/2.7/bin/normalize_by_copy_number.py", line 4, in <module>

    __import__('pkg_resources').run_script('PICRUSt==1.1.0', 'normalize_by_copy_number.py')

  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 719, in run_script

    self.require(requires)[0].run_script(script_name, ns)

  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1504, in run_script

    exec(code, namespace, namespace)

  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PICRUSt-1.1.0-py2.7.egg/EGG-INFO/scripts/normalize_by_copy_number.py", line 132, in <module>

    main()

  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PICRUSt-1.1.0-py2.7.egg/EGG-INFO/scripts/normalize_by_copy_number.py", line 78, in main

    count_table_fh = gzip.open(input_count_table,'rb')

  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gzip.py", line 34, in open

    return GzipFile(filename, mode, compresslevel)

  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gzip.py", line 94, in __init__

    fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb')

IOError: [Errno 2] No such file or directory: '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PICRUSt-1.1.0-py2.7.egg/picrust/data/16S_13_5_precalculated.tab.gz'

MacQIIME nutritions-MacBook-Air:picrust  $ normalize_by_copy_number.py -i otus.biom -o normalized_otus.biom

Traceback (most recent call last):

  File "/Library/Frameworks/Python.framework/Versions/2.7/bin/normalize_by_copy_number.py", line 4, in <module>

    __import__('pkg_resources').run_script('PICRUSt==1.1.0', 'normalize_by_copy_number.py')

  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 719, in run_script

    self.require(requires)[0].run_script(script_name, ns)

  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1504, in run_script

    exec(code, namespace, namespace)

  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PICRUSt-1.1.0-py2.7.egg/EGG-INFO/scripts/normalize_by_copy_number.py", line 132, in <module>

    main()

  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PICRUSt-1.1.0-py2.7.egg/EGG-INFO/scripts/normalize_by_copy_number.py", line 78, in main

    count_table_fh = gzip.open(input_count_table,'rb')

  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gzip.py", line 34, in open

    return GzipFile(filename, mode, compresslevel)

  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gzip.py", line 94, in __init__

    fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb')

IOError: [Errno 2] No such file or directory: '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PICRUSt-1.1.0-py2.7.egg/picrust/data/16S_13_5_precalculated.tab.gz'

Regards,

Austin Ly 

[Gavin Douglas]

Hey Austin,

Sorry you're having problems. I'm not sure why you're getting that error since the precalculated files installed in the correct folder on my mac.

A work around might be to copy the precalculated files to /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PICRUSt-1.1.0-py2.7.egg/picrust/data/

or you could make symbolic links:

cd /path/to/picrust/data/

ln -s $PWD/16S_13_5_precalculated.tab.gz /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PICRUSt-1.1.0-py2.7.egg/picrust/data/16S_13_5_precalculated.tab.gz

ln -s $PWD/ko_13_5_precalculated.tab.gz /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PICRUSt-1.1.0-py2.7.egg/picrust/data/ko_13_5_precalculated.tab.gz

It's likely that there are other issues too if the install didn't finish properly, but you can at least try this!

Gavin

[Victoria Byrd]

Hi Gavin,

I'm having a similar issue. I tried running normalize_by_copy_number.py and got an error:

MacQIIME lri-106100:FilteredMin1000Reads $ normalize_by_copy_number.py -i closed_otu_table_json.biom -o final_otu_table

Traceback (most recent call last):

  File "/usr/local/bin/normalize_by_copy_number.py", line 5, in <module>

    pkg_resources.run_script('PICRUSt==1.1.0', 'normalize_by_copy_number.py')

  File "/Library/Python/2.7/site-packages/pkg_resources/__init__.py", line 735, in run_script

    self.require(requires)[0].run_script(script_name, ns)

  File "/Library/Python/2.7/site-packages/pkg_resources/__init__.py", line 1652, in run_script

    exec(code, namespace, namespace)

  File "/Library/Python/2.7/site-packages/PICRUSt-1.1.0-py2.7.egg/EGG-INFO/scripts/normalize_by_copy_number.py", line 132, in <module>

    main()

  File "/Library/Python/2.7/site-packages/PICRUSt-1.1.0-py2.7.egg/EGG-INFO/scripts/normalize_by_copy_number.py", line 78, in main

    count_table_fh = gzip.open(input_count_table,'rb')

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gzip.py", line 34, in open

    return GzipFile(filename, mode, compresslevel)

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gzip.py", line 94, in __init__

    fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb')

IOError: [Errno 2] No such file or directory: '/Library/Python/2.7/site-packages/PICRUSt-1.1.0-py2.7.egg/picrust/data/16S_13_5_precalculated.tab.gz'

When I tried to find the data directory so I could copy the precalculated files in, as you suggested for Austin, it didn't exist. Nor was I able to create the directory:

MacQIIME lri-106100:FilteredMin1000Reads $ cd /Library/Python/2.7/site-packages/PICRUSt-1.1.0-py2.7.egg/picrust/

MacQIIME lri-106100:picrust $ ls

__init__.py                        make_cluster_jobs.pyc

__init__.pyc                       make_test_datasets.py

ace.py                             make_test_datasets.pyc

ace.pyc                            metagenome_contributions.py

ancestral_state_reconstruction.py  metagenome_contributions.pyc

ancestral_state_reconstruction.pyc parallel.py

bayestraits.py                     parallel.pyc

bayestraits.pyc                    parse.py

count.py                           parse.pyc

count.pyc                          predict_metagenomes.py

evaluate_test_datasets.py          predict_metagenomes.pyc

evaluate_test_datasets.pyc         predict_traits.py

format.py                          predict_traits.pyc

format.pyc                         support_files

format_tree_and_trait_table.py     util.py

format_tree_and_trait_table.pyc    util.pyc

make_cluster_jobs.py

MacQIIME lri-106100:picrust $ mkdir data

mkdir: data: Permission denied

I thought I would try to uninstall and reinstall Picrust 1.1.0 since you stated "it's likely that there are other issues too if the install didn't finish properly" but I still got the same error message. However, I don't think the uninstall and reinstall worked properly, as I got messages like the following one. 

lri-106100:picrust englab$ pip uninstall picrust

Uninstalling PICRUSt-1.1.0:

  /Library/Python/2.7/site-packages/PICRUSt-1.1.0-py2.7.egg

  /usr/local/bin/ancestral_state_reconstruction.py

  /usr/local/bin/categorize_by_function.py

  /usr/local/bin/compare_biom.py

  /usr/local/bin/evaluate_test_datasets.py

  /usr/local/bin/format_tree_and_trait_table.py

  /usr/local/bin/make_test_datasets.py

  /usr/local/bin/metagenome_contributions.py

  /usr/local/bin/normalize_by_copy_number.py

  /usr/local/bin/parallel_predict_traits.py

  /usr/local/bin/pool_test_datasets.py

  /usr/local/bin/predict_metagenomes.py

  /usr/local/bin/predict_traits.py

  /usr/local/bin/print_picrust_config.py

  /usr/local/bin/run_genome_evaluations.py

  /usr/local/bin/scale_metagenome.py

  /usr/local/bin/start_parallel_jobs.py

  /usr/local/bin/start_parallel_jobs_sge.py

  /usr/local/bin/start_parallel_jobs_torque.py

Proceed (y/n)? y

Exception:

Traceback (most recent call last):

  File "/Library/Python/2.7/site-packages/pip/basecommand.py", line 211, in main

    status = self.run(options, args)

  File "/Library/Python/2.7/site-packages/pip/commands/uninstall.py", line 76, in run

    requirement_set.uninstall(auto_confirm=options.yes)

  File "/Library/Python/2.7/site-packages/pip/req/req_set.py", line 305, in uninstall

    req.uninstall(auto_confirm=auto_confirm)

  File "/Library/Python/2.7/site-packages/pip/req/req_install.py", line 716, in uninstall

    paths_to_remove.remove(auto_confirm)

  File "/Library/Python/2.7/site-packages/pip/req/req_uninstall.py", line 125, in remove

    renames(path, new_path)

  File "/Library/Python/2.7/site-packages/pip/utils/__init__.py", line 315, in renames

    shutil.move(old, new)

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 300, in move

    rmtree(src)

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 247, in rmtree

    rmtree(fullname, ignore_errors, onerror)

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 252, in rmtree

    onerror(os.remove, fullname, sys.exc_info())

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 250, in rmtree

    os.remove(fullname)

OSError: [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/PICRUSt-1.1.0-py2.7.egg/EGG-INFO/dependency_links.txt'

You are using pip version 7.1.2, however version 8.1.2 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

lri-106100:picrust englab$ pip install picrust

Requirement already satisfied (use --upgrade to upgrade): picrust in /Library/Python/2.7/site-packages/PICRUSt-1.1.0-py2.7.egg

Requirement already satisfied (use --upgrade to upgrade): numpy>=1.5.1 in /Library/Python/2.7/site-packages (from picrust)

Requirement already satisfied (use --upgrade to upgrade): cogent==1.5.3 in /Library/Python/2.7/site-packages/cogent-1.5.3-py2.7-macosx-10.10-intel.egg (from picrust)

Requirement already satisfied (use --upgrade to upgrade): biom-format<2.2.0,>=2.1.4 in /Library/Python/2.7/site-packages (from picrust)

Requirement already satisfied (use --upgrade to upgrade): pyqi==0.3.2 in /Library/Python/2.7/site-packages (from biom-format<2.2.0,>=2.1.4->picrust)

Requirement already satisfied (use --upgrade to upgrade): scipy>=0.13.0 in /Library/Python/2.7/site-packages (from biom-format<2.2.0,>=2.1.4->picrust)

I should note that I tried uninstalling and reinstalling multiple times, which in retrospect I probably should not have done (I should have asked for help sooner--hopefully I didn't complicate the problem further with my attempts to fix it). On one of my uninstall/reinstall attempts prior to the one I documented above, I got several warning messages upon trying to re-install: 

Installing start_parallel_jobs_torque.py script to /macqiime/anaconda/bin

Installed /macqiime/anaconda/lib/python2.7/site-packages/PICRUSt-1.1.0-py2.7.egg

Processing dependencies for PICRUSt==1.1.0

Searching for biom-format<2.2.0,>=2.1.4

Reading https://pypi.python.org/simple/biom-format/

Best match: biom-format 2.1.5

Downloading https://pypi.python.org/packages/e8/74/9b7e50c29ba548cd53eb97fb9cc1f05a2cf53aa179a64f36e93282b84f79/biom-format-2.1.5.tar.gz#md5=30964772b5eb9a84d407d1ed33a064c0

Processing biom-format-2.1.5.tar.gz

Writing /tmp/easy_install-OtDpbL/biom-format-2.1.5/setup.cfg

Running biom-format-2.1.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-OtDpbL/biom-format-2.1.5/egg-dist-tmp-TlqZrN

no previously-included directories found matching 'docs/_build'

warning: no previously-included files matching '*.pyc' found anywhere in distribution

warning: no previously-included files matching '*.pyo' found anywhere in distribution

warning: no previously-included files matching '.git' found anywhere in distribution

In file included from biom/_filter.c:258:

In file included from /macqiime/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4:

In file included from /macqiime/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17:

In file included from /macqiime/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1804:

/macqiime/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: 

      "Using deprecated NumPy API, disable it by "          "#defining

      NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]

#warning "Using deprecated NumPy API, disable it by " \

 ^

In file included from biom/_filter.c:258:

In file included from /macqiime/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4:

In file included from /macqiime/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:26:

/macqiime/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/__multiarray_api.h:1629:1: warning: 

      unused function '_import_array' [-Wunused-function]

_import_array(void)

^

In file included from biom/_filter.c:259:

In file included from /macqiime/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/ufuncobject.h:317:

/macqiime/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/__ufunc_api.h:241:1: warning: 

      unused function '_import_umath' [-Wunused-function]

_import_umath(void)

^

biom/_filter.c:8085:28: warning: unused function '__Pyx_PyObject_AsString'

      [-Wunused-function]

static CYTHON_INLINE char* __Pyx_PyObject_AsString(PyObject* o) {

                           ^

biom/_filter.c:8082:32: warning: unused function '__Pyx_PyUnicode_FromString'

      [-Wunused-function]

static CYTHON_INLINE PyObject* __Pyx_PyUnicode_FromString(const char* c_str) {

                               ^

.... etc. This went on for a long time. I'm not sure at this point as to what my next step should be. Any suggestions would be very greatly appreciated. Thank you!

[Gavin Douglas]

Hey Victoria,

One problem seems to be that you don't have permission to access some of the library files. You can get around this by typing "sudo" before the uninstall and install commands.

Gavin

[Victoria Byrd]

Gavin, I successfully uninstalled PICRUSt by typing "sudo" as you suggested, and made sure to exit MacQIIME before re-installing PICRUSt because that's what it says to do in the Introduction to QIIME Github book. However, I still get the same error message (IOError: [Errno 2] No such file or directory: '/Library/Python/2.7/site-packages/PICRUSt-1.1.0-py2.7.egg/picrust/data/16S_13_5_precalculated.tab.gz') when I try to run normalize_by_copy_number.py.

Here was my process of uninstalling and re-installing (in case I'm going wrong somewhere with that process):

 sudo pip uninstall picrust

The directory '/Users/englab/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

You are using pip version 7.0.1, however version 8.1.2 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

The directory '/Users/englab/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

Uninstalling PICRUSt-1.1.0:

  /macqiime/anaconda/lib/python2.7/site-packages/PICRUSt-1.1.0-py2.7.egg

Proceed (y/n)? y

  Successfully uninstalled PICRUSt-1.1.0

MacQIIME lri-106100:FilteredMin1000Reads $ cd /Users/englab/opt/picrust-1.1.0

MacQIIME lri-106100:picrust-1.1.0 $ exit

exit

Forgetting MacQIIME environment variable settings...

lri-106100:picrust englab$ pwd

/Users/englab/opt/picrust-1.1.0/picrust

lri-106100:picrust englab$ cd ..

lri-106100:picrust-1.1.0 englab$ sudo python setup.py install

running install

...etc., etc.

Despite re-installing, I checked, and I am indeed still missing the data directory (which should contain the precalculated files) in my Library/Python/2.7/site-packages/PICRUSt-1.1.0-py2.7.egg/picrust/ directory, just like the error message says:

lri-106100:picrust englab$ ls

__init__.py make_cluster_jobs.pyc

__init__.pyc make_test_datasets.py

ace.py make_test_datasets.pyc

ace.pyc metagenome_contributions.py

ancestral_state_reconstruction.py metagenome_contributions.pyc

ancestral_state_reconstruction.pyc parallel.py

bayestraits.py parallel.pyc

bayestraits.pyc parse.py

count.py parse.pyc

count.pyc predict_metagenomes.py

evaluate_test_datasets.py predict_metagenomes.pyc

evaluate_test_datasets.pyc predict_traits.py

format.py predict_traits.pyc

format.pyc support_files

format_tree_and_trait_table.py util.py

format_tree_and_trait_table.pyc util.pyc

make_cluster_jobs.py

Any thoughts? Thanks for your help so far! Victoria

[Gavin Douglas]

Hi Victoria,

I'm not sure if pip will properly uninstall picrust or not, so that's one possible issue.

However, I think the problem is that you're not putting the precalculated files in the data folder before running setup.py, as described here: http://picrust.github.io/picrust/install.html#install

Let me know if that solves it.

Gavin

[Victoria Byrd]

Thanks so much for all your help Gavin,

I am not sure how to properly uninstall if not by using pip (I'm truly a complete newbie to all things bioinformatics, and informatics in general, for that matter). Either way, your second statement is definitely correct--I just realized I was not putting the precalculated folders into the data folder each time I uninstalled and reinstalled picrust (I only did so the first time I installed it). Thanks for pointing that out!

I just saw that picrust is now available in Galaxy, so I may try to use it online for now rather than continue to struggle with the logistics of installation. I really appreciate all your help, though! Thanks again,

Victoria

[Victoria Byrd]

Online version also not working for me-- normalize_by_copy_number.py asking me to check if my file format is .biom (it is). So I tried installing locally again, and downloading the precalculated files into data folder, but the download keeps timing out. Is there a reason for this that you're aware of? The download worked fine last time. I tried clicking on the precalculated file download from the link you sent me, and also downloading from the terminal (based on instructions from the Github book), and neither worked.

Thanks, and sorry for the ton of questions!

[Gavin Douglas]

Hey Victoria,

I would suggest you focus on getting your data in the right format for the PICRUSt galaxy instance since that will probably be easier in the long run. You could make sure you have limited your biom file to "closed OTUs", which will required an extra step if you used an open-reference OTU picking approach (see #1 here: https://github.com/mlangill/microbiome_helper/wiki/PICRUSt-workflow) 

Make sure when you upload the file to the galaxy server that you select "picrust" as the file format. 

If that doesn't work then you can send your biom file privately to me and I can help you troubleshoot. You can email me at: gavin DOT douglas AT dal DOT ca

Gavin