CRITICAL ERROR: Unable to process all thread commands.

1304 views
Skip to first unread message

yun li

unread,
Mar 24, 2016, 11:32:46 AM3/24/16
to HUMAnN Users
Hi Eric,
I met problem again. I tried to run humann2 in parallel. It sucessfully return gene families table. However, error occurs after that.

I receive the following error:

CRITICAL ERROR: Unable to process all thread commands.
Error message returned from command for thread task 2: /appl/python-2.7.10/bin/pytho
n /home/yunli1/.local/lib/python2.7/site-packages/humann2-0.6.2-py2.7.egg/humann2/qu
antify/MinPath12hmp.py -any ...........

Here is the code I used:

#!/bin/bash
#BSUB -J humann2_010_job
#BSUB -o log_humann2_010_%J.out
#BSUB -e log_humann2_010_%J.err

s=(`ls -s *.fastq | sed s/_pe_[1,2].fastq$// | awk '{print $2}'|awk -F\/ '{print $6}'| uniq`)

inputPath=test/
mergedPath=merged_fastq/
outputPath=humann2/

N=5
i=0

(
for filePrefix in ${s[*]:0:10}
    do
        #((i=i%N)); ((i++==0)) && wait

        (cat ${inputPath}${filePrefix}_pe_1.fastq ${inputPath}${filePrefix}_pe_2.fas
tq > ${mergedPath}${filePrefix}_merged.fastq
        humann2 --input ${mergedPath}${filePrefix}_merged.fastq --output humann2/ --taxonomic-profile metaphlan/max_taxonomic_profile.tsv --remove-temp-output && rm ${mergedPath}${filePrefix}_me
rged.fastq ) &
    done
)

yunli...@gmail.com

unread,
Mar 25, 2016, 1:49:23 PM3/25/16
to HUMAnN Users
It looks like the error happened when run "minpath12hmp.py" for one of the tmp file. Here is the error message. Why no problem occurs if I run them one by one?

Error when running glpsol from MinPath.

Error message returned from glpsol :
load_mps: reading LP data from `/basePath/UserName/basePathName/humann2_test/NG-seq_humann2_tmp/tmpgTUqq8/tmpvXJIAB'...
load_mps: name `PATH'
load_mps: 1 rows
load_mps: 0 columns
load_mps: 0 non-zeros
load_mps: 0 right-hand side vector(s)
load_mps: 0 range vector(s)
load_mps: 0 bound vector(s)
load_mps: 7 cards were read
lpx_add_cols: ncs = 0; invalid number of columns

CRITICAL ERROR: Unable to process all thread commands.

Error message returned from command for thread task 16: /appl/python-2.7.10/bin/python /home/userName/.local/lib/python2.7/site-packages/humann2-0.6.2-py2.7.egg/humann2/quantify/MinPath12hmp.py -any /basePath/UserName/basePathName/humann2_test/NG-seq_humann2_tmp/tmpgTUqq8/tmpP2wGTu -map /basePath/UserName/basePathName/humann2_test/NG-seq_humann2_tmp/tmpgTUqq8/tmpQmT_iI -report /basePath/UserName/basePathName/humann2_test/NG-seq_humann2_tmp/tmpgTUqq8/tmpRnUx6v -details /basePath/UserName/basePathName/humann2_test/NG-seq_humann2_tmp/tmpgTUqq8/tmpKmhVJ4 -mps /basePath/UserName/basePathName/humann2_test/NG-seq_humann2_tmp/tmpgTUqq8/tmpvXJIAB

Error when running glpsol from MinPath.

Error message returned from glpsol :
load_mps: reading LP data from `/basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmplWQRJX'...
load_mps: name `PATH'
load_mps: 1 rows
load_mps: 0 columns
load_mps: 0 non-zeros
load_mps: 0 right-hand side vector(s)
load_mps: 0 range vector(s)
load_mps: 0 bound vector(s)
load_mps: 7 cards were read
lpx_add_cols: ncs = 0; invalid number of columns


Error when running glpsol from MinPath.

Error message returned from glpsol :
load_mps: reading LP data from `/basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmpxIg38W'...
load_mps: name `PATH'
load_mps: 1 rows
load_mps: 0 columns
load_mps: 0 non-zeros
load_mps: 0 right-hand side vector(s)
load_mps: 0 range vector(s)
load_mps: 0 bound vector(s)
load_mps: 7 cards were read
lpx_add_cols: ncs = 0; invalid number of columns

CRITICAL ERROR: Unable to process all thread commands.

Error message returned from command for thread task 2: /appl/python-2.7.10/bin/python /home/userName/.local/lib/python2.7/site-packages/humann2-0.6.2-py2.7.egg/humann2/quantify/MinPath12hmp.py -any /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmpZSYEPh -map /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmpydbVeY -report /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmpm_XG54 -details /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmp3BUyCm -mps /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmpxIg38W

Error message returned from command for thread task 3: /appl/python-2.7.10/bin/python /home/userName/.local/lib/python2.7/site-packages/humann2-0.6.2-py2.7.egg/humann2/quantify/MinPath12hmp.py -any /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmpxBwKZl -map /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmpydbVeY -report /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmpEDq77h -details /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmpMeuPEa -mps /basePath/UserName/basePathName/humann2_test/NG-seq2_humann2_temp/tmpyYkzAQ/tmplWQRJX

yunli...@gmail.com

unread,
Mar 29, 2016, 2:07:35 PM3/29/16
to HUMAnN Users, yunli...@gmail.com
Hi,
Any one knows what is wrong with this issue? I am trying to read the source code of humann2. It looks like this error comes from humann2/utilites.py. I tried to find where in humann2 invoke the function "command_threading". however, I can not find in anywhere else that mention "command_threading".
My guess is when more than one humann2 are running at the same time. the command "MinPath12hmp.py" runes before the input temporary file generated?

Thanks,
Yun

Lauren McIver

unread,
Mar 29, 2016, 2:33:03 PM3/29/16
to HUMAnN Users, yunli...@gmail.com
Hi Yun,

I believe the error you are seeing is from glpk which is called by MinPath. Upgrading to a newer version of glpk should resolve the error message. I recommend you reinstall humann2 as this should also install a newer version of glpk by default. 

When you originally installed humann2 did you add the option to bypass install of dependencies? This would also bypass the install of a newer version of glpk. If you have diamond and bowtie2 installed (in a location in your $PATH), you can install humann2 with the default options and it will not install new versions of these tools unless you force an install with the option to replace dependencies installed. By default humann2 will install all dependencies that are not already installed.

Please let me know if you have any questions.

Thank you,
Lauren

yunli...@gmail.com

unread,
Mar 29, 2016, 3:03:16 PM3/29/16
to HUMAnN Users, yunli...@gmail.com
Hi Lauren,
is it like the glpk will be automatically downloaded during the installation or it will included in the humann2 install package. I am running humann2 in a server that can not have web access to the outside. So if in the first case, I should download glpk individually. Could you please let me know the which is the newest version of glpk? I am using humann2-0.6.2.

Thank,
Yun

Lauren McIver

unread,
Mar 29, 2016, 3:27:23 PM3/29/16
to HUMAnN Users
Hi Yun,

Yes, humann2 will need internet access to download glpk prior to install. If your server does not have internet access you could install glpk manually. Here are the steps to install manually (the same version of glpk as humann2 installs v4.55).

1. Download glpk on another machine (and then transfer)

2. Install glpk (replace $PREFIX with the location you want to install glpk, this option is only needed if not installing glpk as root)
$ tar zxvf glpk-4.55.tar.gz
$ cd glpk-4.55
$ ./configure --prefix $PREFIX
$ make
$ make install

3. Check the executable was installed in your path
$ which glpsol

4. Check the install runs
$ glpsol --help

Thank you,
Lauren


yunli...@gmail.com

unread,
Mar 29, 2016, 4:00:25 PM3/29/16
to HUMAnN Users
I downloaded the latest version minpath v1.2 from minpath web site, and put the whole folder of "MinPath" into "humann2/quantify". I saw a folder "glpk-4.6" inside. If I run "./configure --prefix path" inside of "humann2/quantify/glpk-4.6" I will get the following error:

"configure: error: installation or configuration problem: C compiler cannot create executables.
"

Also, I feel confused now. I do not get any error when I run humann2 for one single job, the error happens only when I run more than one job at a time. Since I do not install glpk properly, why I can run humann2 without any error as long as I do not do parallel computing.

Thanks.
Yun

Lauren McIver

unread,
Mar 29, 2016, 4:48:52 PM3/29/16
to HUMAnN Users
Hi Yun,

There is no need to update the MinPath install. If you upgrade the glpk install, you will see the error resolve. Please see my instructions in the prior email (for manual install since your server does not have internet access) and let me know if you have any questions. Hopefully the upgrade should only take you a couple minutes. Since the error is from glpk, running humann2 in serial mode you could still get the error. 

Thank you,
Lauren

yunli...@gmail.com

unread,
Mar 30, 2016, 9:59:30 AM3/30/16
to HUMAnN Users
I did not update the MinPath. There is a glpk folder(glpk-4.6) inside Minpath. I did't run ./configure inside of glpk folder before. However,I did't met any problem when running humann2 sequentially. according to your advise, I tried to run "./configure --prefix path" inside of "humann2/quantify/glpk-4.6" I will get the following error:

"configure: error: installation or configuration problem: C compiler cannot create executables.
"

My question now is:
1. Is glpk package only be used when I run several parallel jobs of humann2?
2. why I can still run humann2 sequentially without glpk installed properly?

Lauren McIver

unread,
Mar 30, 2016, 2:58:51 PM3/30/16
to HUMAnN Users
Hi Yun,

I suggest installing a newer version of glpk. You will want to download the new source and then install. The glpk installed in the MinPath folder is an older version. Please see the instructions that follow (for manual install since your server does not have internet access). If you see errors when running configure for the newer version, I would suggest checking if you have a C compiler installed (ie "gcc --version").

1. Download glpk on another machine (and then transfer to your server)

2. Install glpk (replace $PREFIX with the location you want to install glpk, this option is only needed if not installing glpk as root)
$ tar zxvf glpk-4.55.tar.gz
$ cd glpk-4.55
$ ./configure --prefix $PREFIX
$ make
$ make install

3. Check the executable was installed in your path
$ which glpsol

4. Check the install runs
$ glpsol --help

The MinPath/glpk package is used with all humann2 runs. You currently have an older version of glpk installed with MinPath. 

Please feel free to email me directly if you would like me to help debug manually installing glpk on your server.

Thank you,
Lauren

florentinc...@gmail.com

unread,
Feb 22, 2017, 2:59:07 AM2/22/17
to HUMAnN Users
Hi all,

I am analyzing some metagenomic and metatranscriptomic data.

I started by generating taxonomic profile using metaphlan2 and then I use humann2 command providing the taxonomic profile :

humann2 \
--input $MOD".gz" \
--output humann2 \
--memory-use maximum \
--threads $NPROC \
--gap-fill off \
--prescreen-threshold 0.0001 \
--taxonomic-profile metaphlan2/tuned2/$NAME"_profile.txt" \
--output-format tsv

It worked well on 23 of the 30 metagenomes I analyzed, for the 7 others I got this error:

Computing gene families ...

Computing pathways abundance and coverage ...


Error when running glpsol from MinPath.

Error message returned from glpsol :

load_mps: reading LP data from `/media/florentin/DataDrive05/Flo/Saliva/humann2/4072DNA_R1R2_cat_mod_fastq_humann2_temp/tmp5qnDzi/tmp3AEV3e'...


load_mps: name `PATH'
load_mps: 1 rows
load_mps: 0 columns
load_mps: 0 non-zeros
load_mps: 0 right-hand side vector(s)
load_mps: 0 range vector(s)
load_mps: 0 bound vector(s)
load_mps: 7 cards were read
lpx_add_cols: ncs = 0; invalid number of columns

CRITICAL ERROR: Unable to process all thread commands.

Error message returned from command for thread task 2: /usr/bin/python /home/florentin/.linuxbrew/Cellar/humann2/0.9.3/libexec/lib/python2.7/site-packages/humann2/quantify/MinPath12hmp.py -any /media/florentin/DataDrive05/Flo/Saliva/humann2/4072DNA_R1R2_cat_mod_fastq_humann2_temp/tmp5qnDzi/tmpytwysE -map /media/florentin/DataDrive05/Flo/Saliva/humann2/4072DNA_R1R2_cat_mod_fastq_humann2_temp/tmp5qnDzi/tmp4RAXd8 -report /media/florentin/DataDrive05/Flo/Saliva/humann2/4072DNA_R1R2_cat_mod_fastq_humann2_temp/tmp5qnDzi/tmpkmOgrK -details /media/florentin/DataDrive05/Flo/Saliva/humann2/4072DNA_R1R2_cat_mod_fastq_humann2_temp/tmp5qnDzi/tmptc1AUd -mps /media/florentin/DataDrive05/Flo/Saliva/humann2/4072DNA_R1R2_cat_mod_fastq_humann2_temp/tmp5qnDzi/tmp3AEV3e

I run them again but I got the same error.
I do not understand what is going wrong on those particular samples.

Could you help me to debug that?

Thanks a lot.

Best,

Flo

florentinc...@gmail.com

unread,
Feb 22, 2017, 10:10:55 PM2/22/17
to HUMAnN Users, florentinc...@gmail.com
Hi,

I updated humann2, I have tried to update glpk-4.55 as nicely explained but I got this error:
glpsol --help
glpsol: error while loading shared libraries: libglpk.so.36: cannot open shared object file: No such file or directory

I do not understand why I only have the issue for 7 samples and it is running well for the rest...

Thanks a lot for your help.

Flo

Lauren McIver

unread,
Mar 1, 2017, 1:21:20 PM3/1/17
to HUMAnN Users
Hi Flo,

Upgrading your version of glpk should resolve the error you are seeing. The error occurs with older versions of glpk for small input files. These would be created in the humann2 workflow when a species identified has only a few gene families found. 

My suggestion with the shared libraries error would be to try reinstalling glpk again. When you install humann2, it should try to install the newer glpk and fall back to the older glpk install if there is an error building the newer version. You could try reinstalling humann2, adding the option "--replace-dependencies-install", and it will reinstall all dependencies including glpk. The other option would be to add the location of your new glpk install libraries to your $LD_LIBRARY_PATH environment variable. 

Thanks!
Lauren

bob nichols

unread,
Apr 5, 2017, 12:51:14 PM4/5/17
to HUMAnN Users
Hi Lauren,
I have run humann2 very successfully in the past but when I recently updated to version 0.6.1 I got the error message discussed in this thread. I am currently running humann2 on an external server so I do not have root access. I manually installed glpk to version 4.61 but when I went to check the actual version I got this error message. 

glpsol: error while loading shared libraries: libglpk.so.40: cannot open shared object file: No such file or directory

I have tried reinstalling Humann2 but then that just puts me back at square 1 and I get the  "Unable to process all thread commands." error. So I am just curious if you have any more information on how to fix this issue.
Thanks,
Bob 

Lauren McIver

unread,
Apr 5, 2017, 1:17:53 PM4/5/17
to HUMAnN Users
Hi Bob,
My guess is that humann2 does not see your manually installed glpk and that is why it is installing glpk when you upgrade humann2. If humann2 finds the executable glpsol in your path it will not install glpk. Alternatively, if you install humann2 with the option --bypass-dependencies-install it will not install any of the dependencies even if they are not found in your path (ie glpk, diamond, bowtie2). 

It also sounds like humann2 is unable to install a new glpk on your platform so it falls back to installing an older version of glpk. The older glpk version has issues with small input files that causes the errors you are seeing. The new glpk will not be able to be installed if your platform does not have gcc or make installed. If you could install humann2 again adding the verbose flag, we will be able to catch the error messages about the new glpk install. Alternatively we could work to get your current glpk install up and running. Did your LD_LIBRARY_PATH change recently? This could account for the error messages you are seeing. Please feel free to respond to me directly with your specific platform information and I can help you resolve the error message.

Thanks,
Lauren

jincheng...@gmail.com

unread,
May 15, 2018, 10:13:40 AM5/15/18
to HUMAnN Users
Dear all,

I am writing because I just installed the newest version of humann2 using `pip install --user humann2`, but still have this exact error. I thought humann2 will automatically install glpk as stated in the tutorial.
Is there a way to check the version of glpk? Could I reinstall glpk?
Thanks!

Jincheng

Lauren McIver

unread,
May 15, 2018, 6:07:33 PM5/15/18
to HUMAnN Users
Hi Jincheng, HUMAnN2 should by default install glpk. You can check for the install by running "which glpsol". If you do not see this installed then it might be that your pip environment is set to use wheels instead of installing from source. If this is the case then add the options described in the Getting Started with HUMAnN2 section to force pip to install from source: http://huttenhower.sph.harvard.edu/humann2#markdown-header-getting-started-with-humann2

Thanks!
Lauren

jincheng...@gmail.com

unread,
May 17, 2018, 4:34:11 PM5/17/18
to HUMAnN Users
Thanks, Lauren! Yes, forcing pip to install from source instead of wheel did solve the issue.

Martin Cuma

unread,
May 14, 2019, 2:17:04 PM5/14/19
to HUMAnN Users
Hi everyone,

we have an user with the same error, and wondering what we can do about this.

We are using the Docker container from docker://biobakery/workflows, because of the complications associated with installing bioBakery natively on CentOS 7 that runs on our clusters. That container has glpsol version 4.6. Would updating gplsol to even newer version solve this problem? If so, any chance to update the Docker container on Dockerhub?

Thanks,
Martin

Lauren McIver

unread,
May 15, 2019, 3:47:25 PM5/15/19
to HUMAnN Users
Hi Martin, You are right that it does include an older version of glpsol. I have updated the version and posted a new image (tag: v0.1). 

Thanks,
Lauren

--
You received this message because you are subscribed to the Google Groups "HUMAnN Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to humann-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/humann-users/afa018b5-f277-4e0b-a9e0-0f2b81a7d8f6%40googlegroups.com.

Martin Cuma

unread,
May 16, 2019, 12:26:11 AM5/16/19
to HUMAnN Users
Thanks Lauren, that did the trick.

Frederick Shi

unread,
Jun 10, 2019, 12:41:04 PM6/10/19
to HUMAnN Users
Hi Lauren,
I tried to force install humann2 and the glpsol is still not installed. I wonder if you can help.

Thanks,
Fred
Reply all
Reply to author
Forward
0 new messages