CRITICAL ERROR: Unable to process all thread commands.

[yun li]

Hi Eric,

I met problem again. I tried to run humann2 in parallel. It sucessfully return gene families table. However, error occurs after that.

I receive the following error:

CRITICAL ERROR: Unable to process all thread commands.

Error message returned from command for thread task 2: /appl/python-2.7.10/bin/pytho

n /home/yunli1/.local/lib/python2.7/site-packages/humann2-0.6.2-py2.7.egg/humann2/qu

antify/MinPath12hmp.py -any ...........

Here is the code I used:

#!/bin/bash

#BSUB -J humann2_010_job

#BSUB -o log_humann2_010_%J.out

#BSUB -e log_humann2_010_%J.err

s=(`ls -s *.fastq | sed s/_pe_[1,2].fastq$// | awk '{print $2}'|awk -F\/ '{print $6}'| uniq`)

inputPath=test/

mergedPath=merged_fastq/

outputPath=humann2/

N=5

i=0

(

for filePrefix in ${s[*]:0:10}

    do

        #((i=i%N)); ((i++==0)) && wait

        (cat ${inputPath}${filePrefix}_pe_1.fastq ${inputPath}${filePrefix}_pe_2.fas

tq > ${mergedPath}${filePrefix}_merged.fastq

        humann2 --input ${mergedPath}${filePrefix}_merged.fastq --output humann2/ --taxonomic-profile metaphlan/max_taxonomic_profile.tsv --remove-temp-output && rm ${mergedPath}${filePrefix}_me

rged.fastq ) &

    done

)

[yunli...@gmail.com]

It looks like the error happened when run "minpath12hmp.py" for one of the tmp file. Here is the error message. Why no problem occurs if I run them one by one?

Error when running glpsol from MinPath.

Error message returned from glpsol :
load_mps: reading LP data from `/basePath/UserName/basePathName/humann2_test/NG-seq_humann2_tmp/tmpgTUqq8/tmpvXJIAB'...
load_mps: name `PATH'
load_mps: 1 rows
load_mps: 0 columns
load_mps: 0 non-zeros
load_mps: 0 right-hand side vector(s)
load_mps: 0 range vector(s)
load_mps: 0 bound vector(s)
load_mps: 7 cards were read
lpx_add_cols: ncs = 0; invalid number of columns

CRITICAL ERROR: Unable to process all thread commands.

Error message returned from command for thread task 16: /appl/python-2.7.10/bin/python /home/userName/.local/lib/python2.7/site-packages/humann2-0.6.2-py2.7.egg/humann2/quantify/MinPath12hmp.py -any /basePath/UserName/basePathName/humann2_test/NG-seq_humann2_tmp/tmpgTUqq8/tmpP2wGTu -map /basePath/UserName/basePathName/humann2_test/NG-seq_humann2_tmp/tmpgTUqq8/tmpQmT_iI -report /basePath/UserName/basePathName/humann2_test/NG-seq_humann2_tmp/tmpgTUqq8/tmpRnUx6v -details /basePath/UserName/basePathName/humann2_test/NG-seq_humann2_tmp/tmpgTUqq8/tmpKmhVJ4 -mps /basePath/UserName/basePathName/humann2_test/NG-seq_humann2_tmp/tmpgTUqq8/tmpvXJIAB

Error when running glpsol from MinPath.

Error message returned from glpsol :
load_mps: reading LP data from `/basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmplWQRJX'...
load_mps: name `PATH'
load_mps: 1 rows
load_mps: 0 columns
load_mps: 0 non-zeros
load_mps: 0 right-hand side vector(s)
load_mps: 0 range vector(s)
load_mps: 0 bound vector(s)
load_mps: 7 cards were read
lpx_add_cols: ncs = 0; invalid number of columns

Error when running glpsol from MinPath.

Error message returned from glpsol :
load_mps: reading LP data from `/basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmpxIg38W'...
load_mps: name `PATH'
load_mps: 1 rows
load_mps: 0 columns
load_mps: 0 non-zeros
load_mps: 0 right-hand side vector(s)
load_mps: 0 range vector(s)
load_mps: 0 bound vector(s)
load_mps: 7 cards were read
lpx_add_cols: ncs = 0; invalid number of columns

CRITICAL ERROR: Unable to process all thread commands.

Error message returned from command for thread task 2: /appl/python-2.7.10/bin/python /home/userName/.local/lib/python2.7/site-packages/humann2-0.6.2-py2.7.egg/humann2/quantify/MinPath12hmp.py -any /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmpZSYEPh -map /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmpydbVeY -report /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmpm_XG54 -details /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmp3BUyCm -mps /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmpxIg38W

Error message returned from command for thread task 3: /appl/python-2.7.10/bin/python /home/userName/.local/lib/python2.7/site-packages/humann2-0.6.2-py2.7.egg/humann2/quantify/MinPath12hmp.py -any /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmpxBwKZl -map /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmpydbVeY -report /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmpEDq77h -details /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmpMeuPEa -mps /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmplWQRJX

[yunli...@gmail.com]

Hi,
Any one knows what is wrong with this issue?　I am trying to read the source code of humann2. It looks like this error comes from humann2/utilites.py. I tried to find where in humann2 invoke the function "command_threading". however, I can not find in anywhere else that mention "command_threading".
My guess is when more than one humann2 are running at the same time. the command "MinPath12hmp.py" runes before the input temporary file generated?

Thanks,
Yun

[Lauren McIver]

Hi Yun,

I believe the error you are seeing is from glpk which is called by MinPath. Upgrading to a newer version of glpk should resolve the error message. I recommend you reinstall humann2 as this should also install a newer version of glpk by default. 

When you originally installed humann2 did you add the option to bypass install of dependencies? This would also bypass the install of a newer version of glpk. If you have diamond and bowtie2 installed (in a location in your $PATH), you can install humann2 with the default options and it will not install new versions of these tools unless you force an install with the option to replace dependencies installed. By default humann2 will install all dependencies that are not already installed.

Please let me know if you have any questions.

Thank you,

Lauren

[yunli...@gmail.com]

Hi Lauren,
is it like the glpk will be automatically downloaded during the installation or it will included in the humann2 install package. I am running humann2 in a server that can not have web access to the outside. So if in the first case, I should download glpk individually. Could you please let me know the which is the newest version of glpk? I am using humann2-0.6.2.

Thank,
Yun

[Lauren McIver]

Hi Yun,

Yes, humann2 will need internet access to download glpk prior to install. If your server does not have internet access you could install glpk manually. Here are the steps to install manually (the same version of glpk as humann2 installs v4.55).

1. Download glpk on another machine (and then transfer)

$ wget http://ftp.gnu.org/gnu/glpk/glpk-4.55.tar.gz

2. Install glpk (replace $PREFIX with the location you want to install glpk, this option is only needed if not installing glpk as root)

$ tar zxvf glpk-4.55.tar.gz

$ cd glpk-4.55

$ ./configure --prefix $PREFIX

$ make

$ make install

3. Check the executable was installed in your path

$ which glpsol

4. Check the install runs

$ glpsol --help

Thank you,

Lauren

[yunli...@gmail.com]

I downloaded the latest version minpath v1.2 from minpath web site, and put the whole folder of "MinPath" into "humann2/quantify". I saw a folder "glpk-4.6" inside. If I run "./configure --prefix path" inside of "humann2/quantify/glpk-4.6" I will get the following error:

"configure: error: installation or configuration problem: C compiler cannot create executables.
"

Also, I feel confused now. I do not get any error when I run humann2 for one single job, the error happens only when I run more than one job at a time. Since I do not install glpk properly, why I can run humann2 without any error as long as I do not do parallel computing.

Thanks.
Yun

[Lauren McIver]

Hi Yun,

There is no need to update the MinPath install. If you upgrade the glpk install, you will see the error resolve. Please see my instructions in the prior email (for manual install since your server does not have internet access) and let me know if you have any questions. Hopefully the upgrade should only take you a couple minutes. Since the error is from glpk, running humann2 in serial mode you could still get the error. 

Thank you,

Lauren

[yunli...@gmail.com]

I did not update the MinPath. There is a glpk folder(glpk-4.6) inside Minpath. I did't run ./configure inside of glpk folder before. However,I did't met any problem when running humann2 sequentially. according to your advise, I tried to run "./configure --prefix path" inside of "humann2/quantify/glpk-4.6" I will get the following error:

"configure: error: installation or configuration problem: C compiler cannot create executables.
"

My question now is:
1. Is glpk package only be used when I run several parallel jobs of humann2?
2. why I can still run humann2 sequentially without glpk installed properly?

[Lauren McIver]

Hi Yun,

I suggest installing a newer version of glpk. You will want to download the new source and then install. The glpk installed in the MinPath folder is an older version. Please see the instructions that follow (for manual install since your server does not have internet access). If you see errors when running configure for the newer version, I would suggest checking if you have a C compiler installed (ie "gcc --version").

1. Download glpk on another machine (and then transfer to your server)

$ wget http://ftp.gnu.org/gnu/glpk/glpk-4.55.tar.gz

2. Install glpk (replace $PREFIX with the location you want to install glpk, this option is only needed if not installing glpk as root)

$ tar zxvf glpk-4.55.tar.gz

$ cd glpk-4.55

$ ./configure --prefix $PREFIX

$ make

$ make install

3. Check the executable was installed in your path

$ which glpsol

4. Check the install runs

$ glpsol --help

The MinPath/glpk package is used with all humann2 runs. You currently have an older version of glpk installed with MinPath. 

Please feel free to email me directly if you would like me to help debug manually installing glpk on your server.

Thank you,

Lauren

[florentinc...@gmail.com]

Hi all,

I am analyzing some metagenomic and metatranscriptomic data.

I started by generating taxonomic profile using metaphlan2 and then I use humann2 command providing the taxonomic profile :

humann2 \
--input $MOD".gz" \
--output humann2 \
--memory-use maximum \
--threads $NPROC \
--gap-fill off \
--prescreen-threshold 0.0001 \
--taxonomic-profile metaphlan2/tuned2/$NAME"_profile.txt" \
--output-format tsv

It worked well on 23 of the 30 metagenomes I analyzed, for the 7 others I got this error:

Computing gene families ...

Computing pathways abundance and coverage ...

Error when running glpsol from MinPath.

Error message returned from glpsol :

load_mps: reading LP data from `/media/florentin/DataDrive05/Flo/Saliva/humann2/4072DNA_R1R2_cat_mod_fastq_humann2_temp/tmp5qnDzi/tmp3AEV3e'...

load_mps: name `PATH'
load_mps: 1 rows
load_mps: 0 columns
load_mps: 0 non-zeros
load_mps: 0 right-hand side vector(s)
load_mps: 0 range vector(s)
load_mps: 0 bound vector(s)
load_mps: 7 cards were read
lpx_add_cols: ncs = 0; invalid number of columns

CRITICAL ERROR: Unable to process all thread commands.

Error message returned from command for thread task 2: /usr/bin/python /home/florentin/.linuxbrew/Cellar/humann2/0.9.3/libexec/lib/python2.7/site-packages/humann2/quantify/MinPath12hmp.py -any /media/florentin/DataDrive05/Flo/Saliva/humann2/4072DNA_R1R2_cat_mod_fastq_humann2_temp/tmp5qnDzi/tmpytwysE -map /media/florentin/DataDrive05/Flo/Saliva/humann2/4072DNA_R1R2_cat_mod_fastq_humann2_temp/tmp5qnDzi/tmp4RAXd8 -report /media/florentin/DataDrive05/Flo/Saliva/humann2/4072DNA_R1R2_cat_mod_fastq_humann2_temp/tmp5qnDzi/tmpkmOgrK -details /media/florentin/DataDrive05/Flo/Saliva/humann2/4072DNA_R1R2_cat_mod_fastq_humann2_temp/tmp5qnDzi/tmptc1AUd -mps /media/florentin/DataDrive05/Flo/Saliva/humann2/4072DNA_R1R2_cat_mod_fastq_humann2_temp/tmp5qnDzi/tmp3AEV3e

I run them again but I got the same error.
I do not understand what is going wrong on those particular samples.

Could you help me to debug that?

Thanks a lot.

Best,

Flo

[florentinc...@gmail.com]

Hi,

I updated humann2, I have tried to update glpk-4.55 as nicely explained but I got this error:
glpsol --help
glpsol: error while loading shared libraries: libglpk.so.36: cannot open shared object file: No such file or directory

I do not understand why I only have the issue for 7 samples and it is running well for the rest...

Thanks a lot for your help.

Flo

[Lauren McIver]

Hi Flo,

Upgrading your version of glpk should resolve the error you are seeing. The error occurs with older versions of glpk for small input files. These would be created in the humann2 workflow when a species identified has only a few gene families found. 

My suggestion with the shared libraries error would be to try reinstalling glpk again. When you install humann2, it should try to install the newer glpk and fall back to the older glpk install if there is an error building the newer version. You could try reinstalling humann2, adding the option "--replace-dependencies-install", and it will reinstall all dependencies including glpk. The other option would be to add the location of your new glpk install libraries to your $LD_LIBRARY_PATH environment variable. 

Thanks!

Lauren

[bob nichols]

Hi Lauren,

I have run humann2 very successfully in the past but when I recently updated to version 0.6.1 I got the error message discussed in this thread. I am currently running humann2 on an external server so I do not have root access. I manually installed glpk to version 4.61 but when I went to check the actual version I got this error message. 

glpsol: error while loading shared libraries: libglpk.so.40: cannot open shared object file: No such file or directory

I have tried reinstalling Humann2 but then that just puts me back at square 1 and I get the  "Unable to process all thread commands." error. So I am just curious if you have any more information on how to fix this issue.

Thanks,

Bob 

[Lauren McIver]

Hi Bob,

My guess is that humann2 does not see your manually installed glpk and that is why it is installing glpk when you upgrade humann2. If humann2 finds the executable glpsol in your path it will not install glpk. Alternatively, if you install humann2 with the option --bypass-dependencies-install it will not install any of the dependencies even if they are not found in your path (ie glpk, diamond, bowtie2). 

It also sounds like humann2 is unable to install a new glpk on your platform so it falls back to installing an older version of glpk. The older glpk version has issues with small input files that causes the errors you are seeing. The new glpk will not be able to be installed if your platform does not have gcc or make installed. If you could install humann2 again adding the verbose flag, we will be able to catch the error messages about the new glpk install. Alternatively we could work to get your current glpk install up and running. Did your LD_LIBRARY_PATH change recently? This could account for the error messages you are seeing. Please feel free to respond to me directly with your specific platform information and I can help you resolve the error message.

Thanks,

Lauren

[jincheng...@gmail.com]

Dear all,

I am writing because I just installed the newest version of humann2 using `pip install --user humann2`, but still have this exact error. I thought humann2 will automatically install glpk as stated in the tutorial.
Is there a way to check the version of glpk? Could I reinstall glpk?
Thanks!

Jincheng

[Lauren McIver]

Hi Jincheng, HUMAnN2 should by default install glpk. You can check for the install by running "which glpsol". If you do not see this installed then it might be that your pip environment is set to use wheels instead of installing from source. If this is the case then add the options described in the Getting Started with HUMAnN2 section to force pip to install from source: http://huttenhower.sph.harvard.edu/humann2#markdown-header-getting-started-with-humann2 . 

Thanks!

Lauren

[jincheng...@gmail.com]

Thanks, Lauren! Yes, forcing pip to install from source instead of wheel did solve the issue.

[Martin Cuma]

Hi everyone,

we have an user with the same error, and wondering what we can do about this.

We are using the Docker container from docker://biobakery/workflows, because of the complications associated with installing bioBakery natively on CentOS 7 that runs on our clusters. That container has glpsol version 4.6. Would updating gplsol to even newer version solve this problem? If so, any chance to update the Docker container on Dockerhub?

Thanks,

Martin

[Lauren McIver]

Hi Martin, You are right that it does include an older version of glpsol. I have updated the version and posted a new image (tag: v0.1). 

Thanks,

Lauren

--
You received this message because you are subscribed to the Google Groups "HUMAnN Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to humann-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/humann-users/afa018b5-f277-4e0b-a9e0-0f2b81a7d8f6%40googlegroups.com.

[Martin Cuma]

Thanks Lauren, that did the trick.

[Frederick Shi]

Hi Lauren,

I tried to force install humann2 and the glpsol is still not installed. I wonder if you can help.

Thanks,

Fred