Automatic installation fails

[Stephen]

I am trying to get bcbio-nextgen setup on our cluster, but I cannot get it installed. There are actually several errors occurring, none of which I have been able to fix.

I have run all the 'unset PYTHONPATH' commands listed in the docs, and have Python 2.7.3 set as my default.

Here is my install command:

unset PYTHONHOME; unset PYTHONPATH; export PYTHONNOUSERSITE=1
python bcbio_nextgen_install.py data/ --tooldir=tools/ --genomes GRCh37 --aligners bwa --aligners bowtie2 --isolate

And here is some of the output log, I've bolded some of the error messages I found

--2017-05-11 18:40:53--  https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh
Resolving repo.continuum.io... 104.16.18.10, 104.16.19.10, 2400:cb00:2048:1::6810:130a, ...
Connecting to repo.continuum.io|104.16.18.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 28795989 (27M) [application/x-sh]
Saving to: `Miniconda2-latest-Linux-x86_64.sh'

     0K ........ ........ ........ ........ ........ ........ 10% 34.5M 1s
  3072K ........ ........ ........ ........ ........ ........ 21% 44.4M 1s
  6144K ........ ........ ........ ........ ........ ........ 32% 29.9M 1s

...
...
<truncated>
...
...

snpeff:                            4.3i-0                     bioconda
spades:                            3.10.1-py27_0              bioconda
star:                              2.5.3a-0                   bioconda
stringtie:                         1.3.3-py27_1               bioconda
subread:                           1.5.2-0                    bioconda
svtools:                           0.3.2-py27_0               bioconda
svtyper:                           0.1.4-py27_0               bioconda
tbb:                               4.4_20150728-0             bioconda
tdrmapper:                         1.1-2                      bioconda
theta2:                            0.7-py27_0                 bioconda
toolshed:                          0.4.6-py27_0               bioconda
tophat:                            2.1.1-py27_0               bioconda
tophat-recondition:                1.4-py27_0                 bioconda
ucsc-bedg/ifs/home/kellys04/pipeline_manangers/bcbio-nextgen/data/anaconda/bin/.genesplicer-post-link.sh: line 9: warning: here-document at line 3 delimited by end-of-file (wanted `EOF')
It looks like GATK has not yet been installed.

Usage: gatk-register /path/to/GenomeAnalysisTK[-3.7.tar.bz2|.jar]

Due to license restrictions, this recipe cannot distribute
and install GATK directly. To fully install GATK, you must
download a licensed copy of GATK from the Broad Institute:
https://www.broadinstitute.org/gatk/download/
and run (after installing this package):
gatk-register /path/to/GenomeAnalysisTK[-3.7.tar.bz2|.jar],
This will copy GATK into your conda environment.


PaddingError: Placeholder of length '80' too short in package r::r-base-3.2.2-0.
The package must be rebuilt with conda-build > 2.0.


raphtobigwig:             332-0                      bioconda
ucsc-bedtobigbed:                  332-0                      bioconda
ucsc-bigbedinfo:                   332-0                      bioconda
ucsc-bigbedsummary:                332-0                      bioconda
ucsc-bigbedtobed:                  332-0                      bioconda
ucsc-bigwiginfo:                   332-0                      bioconda
ucsc-bigwigsummary:                332-0                      bioconda
ucsc-bigwigtobedgraph:             332-0                      bioconda
ucsc-bigwigtowig:                  332-0                      bioconda
ucsc-fatotwobit:                   332-0                      bioconda
ucsc-gtftogenepred:                332-0                      bioconda
ucsc-liftover:                     324-0                      bioconda
ucsc-wigtobigwig:                  332-0                      bioconda
umis:                              0.6.0a-py27_1              bioconda
unidecode:                         0.04.19-py27_0             conda-forge
unzip:                             6.0-0                      conda-forge
vardict:                           2017.04.18-pl5.22.0_0      bioconda
vardict-java:                      1.5.0-0                    bioconda
variant-effect-predictor:          87-0                       bioconda
variantbam:                        1.3.0-0                    bioconda
varscan:                           2.4.2-0                    bioconda
vawk:                              0.0.2-py27_1               bioconda
vcf2db:                            2017.03.01-py27_0          bioconda
vcfanno:                           0.2.6-0                    bioconda
vcflib:                            1.0.0_rc1-0                bioconda
viennarna:                         2.3.3-0                    bioconda
vt:                                2015.11.10-1               bioconda
wham:                              1.7.0.307-0                bioconda
xopen:                             0.1.1-py27_0               bioconda

The following packages will be DOWNGRADED due to dependency conflicts:

fontconfig:                        2.12.1-4                   conda-forge --> 2.11.1-6          conda-forge
freetype:                          2.7-1                      conda-forge --> 2.6.3-1           conda-forge
icu:                               58.1-1                     conda-forge --> 56.1-4            conda-forge
libxml2:                           2.9.4-4                    conda-forge --> 2.9.4-3           conda-forge
matplotlib:                        2.0.2-np112py27_0          conda-forge --> 2.0.0-np112py27_1 conda-forge
pillow:                            4.1.1-py27_0               conda-forge --> 4.0.0-py27_1      conda-forge
pyqt:                              5.6.0-py27_2               conda-forge --> 4.11.4-py27_2     conda-forge
qt:                                5.6.2-2                    conda-forge --> 4.8.7-3           conda-forge


Fatal error: local() encountered an error (return code 1) while executing '/ifs/home/kellys04/pipeline_manangers/bcbio-nextgen/data/anaconda/bin/conda install --quiet -y -c bioconda -c conda-forge -c r age-metasv bamtools bamutil bcbio-prioritize bcbio-rnaseq bcbio-variation bcbio-variation-recall bcbreport bcftools bedtools=2.26.0 bio-vcf biobambam bowtie bowtie2=2.2.8 bwa bwakit cage cancerit-allelecount cnvkit cpat cramtools crossmap cufflinks cutadapt cyvcf2 delly dkfz-bias-filter ensembl-vep=88.* express extract-sv-reads fastqc fgbio freebayes=1.1.0 gatk gatk-framework gemini geneimpacts genesplicer glpk=4.57 goleft grabix gridss gsort gvcf-regions hap.py hisat2 hmmlearn htseq htslib impute2 kallisto kraken lofreq lumpy-sv macs2 manta maxentscan mbuffer metasv mirdeep2=2.0.0.7 moreutils multiqc multiqc-bcbio ngs-disambiguate novoalign oncofuse optitype pandoc parallel pbgzip perl-sanger-cgp-battenberg phylowgs picard pindel platypus-variant pyloh pytabix pythonpy qsignature qualimap rapmap razers3 rtg-tools sailfish salmon sambamba samblaster samtools scalpel sentieon seq2c seqbuster seqtk sickle-trim simple_sv_annotation snap-aligner=1.0dev.97 snpeff=4.3i spades star stringtie subread svtools svtyper tdrmapper theta2 tophat tophat-recondition ucsc-bedgraphtobigwig ucsc-bedtobigbed ucsc-bigbedinfo ucsc-bigbedsummary ucsc-bigbedtobed ucsc-bigwiginfo ucsc-bigwigsummary ucsc-bigwigtobedgraph ucsc-bigwigtowig ucsc-fatotwobit ucsc-gtftogenepred ucsc-liftover ucsc-wigtobigwig umis vardict vardict-java variant-effect-predictor=87 variantbam varscan vawk vcf2db vcfanno vcflib viennarna vt wham anaconda-client awscli nodejs p7zip s3gof3r xz perl-app-cpanminus perl-archive-extract perl-archive-zip perl-bio-db-sam perl-cgi perl-dbi perl-encode-locale perl-file-fetch perl-file-sharedir perl-file-sharedir-install perl-ipc-system-simple perl-lwp-protocol-https perl-lwp-simple perl-statistics-descriptive perl-time-hires perl-vcftools-vcf bioconductor-annotate bioconductor-biocgenerics=0.18.0 bioconductor-biocinstaller bioconductor-biocstyle bioconductor-biostrings bioconductor-biovizbase bioconductor-bubbletree bioconductor-cn.mops bioconductor-degreport bioconductor-deseq2 bioconductor-dexseq bioconductor-dnacopy bioconductor-genomicranges bioconductor-iranges=2.4.7 bioconductor-limma bioconductor-rtracklayer r r-cghflasso r-chbutils r-devtools r-dplyr r-dt r-ggdendro r-ggplot2 r-gplots r-gsalib r-knitr r-pheatmap r-plyr r-pscbs r-reshape r-rmarkdown r-rsqlite r-sleuth r-snow r-stringi r-wasabi'

Aborting.
Upgrading bcbio
Upgrading third party tools to latest versions
Setting up virtual machine
[localhost] local: echo $HOME
[localhost] local: uname -m
[localhost] local: /ifs/home/kellys04/pipeline_manangers/bcbio-nextgen/data/anaconda/bin/conda info --json
[localhost] local: /ifs/home/kellys04/pipeline_manangers/bcbio-nextgen/data/anaconda/bin/conda install --quiet -y -c bioconda -c conda-forge -c r age-metasv bamtools bamutil bcbio-prioritize bcbio-rnaseq bcbio-variation bcbio-variation-recall bcbreport bcftools bedtools=2.26.0 bio-vcf biobambam bowtie bowtie2=2.2.8 bwa bwakit cage cancerit-allelecount cnvkit cpat cramtools crossmap cufflinks cutadapt cyvcf2 delly dkfz-bias-filter ensembl-vep=88.* express extract-sv-reads fastqc fgbio freebayes=1.1.0 gatk gatk-framework gemini geneimpacts genesplicer glpk=4.57 goleft grabix gridss gsort gvcf-regions hap.py hisat2 hmmlearn htseq htslib impute2 kallisto kraken lofreq lumpy-sv macs2 manta maxentscan mbuffer metasv mirdeep2=2.0.0.7 moreutils multiqc multiqc-bcbio ngs-disambiguate novoalign oncofuse optitype pandoc parallel pbgzip perl-sanger-cgp-battenberg phylowgs picard pindel platypus-variant pyloh pytabix pythonpy qsignature qualimap rapmap razers3 rtg-tools sailfish salmon sambamba samblaster samtools scalpel sentieon seq2c seqbuster seqtk sickle-trim simple_sv_annotation snap-aligner=1.0dev.97 snpeff=4.3i spades star stringtie subread svtools svtyper tdrmapper theta2 tophat tophat-recondition ucsc-bedgraphtobigwig ucsc-bedtobigbed ucsc-bigbedinfo ucsc-bigbedsummary ucsc-bigbedtobed ucsc-bigwiginfo ucsc-bigwigsummary ucsc-bigwigtobedgraph ucsc-bigwigtowig ucsc-fatotwobit ucsc-gtftogenepred ucsc-liftover ucsc-wigtobigwig umis vardict vardict-java variant-effect-predictor=87 variantbam varscan vawk vcf2db vcfanno vcflib viennarna vt wham anaconda-client awscli nodejs p7zip s3gof3r xz perl-app-cpanminus perl-archive-extract perl-archive-zip perl-bio-db-sam perl-cgi perl-dbi perl-encode-locale perl-file-fetch perl-file-sharedir perl-file-sharedir-install perl-ipc-system-simple perl-lwp-protocol-https perl-lwp-simple perl-statistics-descriptive perl-time-hires perl-vcftools-vcf bioconductor-annotate bioconductor-biocgenerics=0.18.0 bioconductor-biocinstaller bioconductor-biocstyle bioconductor-biostrings bioconductor-biovizbase bioconductor-bubbletree bioconductor-cn.mops bioconductor-degreport bioconductor-deseq2 bioconductor-dexseq bioconductor-dnacopy bioconductor-genomicranges bioconductor-iranges=2.4.7 bioconductor-limma bioconductor-rtracklayer r r-cghflasso r-chbutils r-devtools r-dplyr r-dt r-ggdendro r-ggplot2 r-gplots r-gsalib r-knitr r-pheatmap r-plyr r-pscbs r-reshape r-rmarkdown r-rsqlite r-sleuth r-snow r-stringi r-wasabi
Checking required dependencies
Installing isolated base python installation
Installing bcbio-nextgen
Installing data and third party dependencies
Traceback (most recent call last):
File "bcbio_nextgen_install.py", line 251, in <module>
main(parser.parse_args(), sys.argv[1:])
File "bcbio_nextgen_install.py", line 44, in main
subprocess.check_call([bcbio, "upgrade"] + _clean_args(sys_argv, args))
File "/local/apps/python/2.7.3/lib/python2.7/subprocess.py", line 511, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/ifs/home/kellys04/pipeline_manangers/bcbio-nextgen/data/anaconda/bin/bcbio_nextgen.py', 'upgrade', '--tooldir=tools/', '--genomes', 'GRCh37', '--aligners', 'bwa', '--aligners', 'bowtie2', '--isolate', '--data']' returned non-zero exit status 1

When I try to fix the GATK installation with the supplied command, I get this error:

$ gatk-register ~/software/GenomeAnalysisTK-3.7.tar.bz2
bash: gatk-register: command not found

Any idea how to fix this? Since the entire setup process is completely automated, it is not clear where to even start in trying to get this working.

[Stephen]

Also this is running on CentOS 6

[Brad Chapman]

Stephen;
Sorry about the install issues, and thank you for reporting. It looks like the
error is:

> *PaddingError: Placeholder of length '80' too short in package
> r::r-base-3.2.2-0.The package must be rebuilt with conda-build > 2.0.*

The path to your bcbio install:

/ifs/home/kellys04/pipeline_manangers/bcbio-nextgen/data/anaconda

is a bit too long for this older package. You may be able to work around
this in the term by installing to a short path (maybe just `bcbio`
insteadl of `bcbio-nextgen`, or avoiding the `pipeline_managers` folder).
There is more detail on the error here:

https://github.com/ContinuumIO/anaconda-recipes/issues/88

Sorry about the hack. We're in the process of migrating to R 3.4 so should
have new packages without this limitation soon.

The GATK reports are just warnings from the gatk tool, more info on
registering a GATK jar, if you need it after the install, is here:
http://bcbio-nextgen.readthedocs.io/en/latest/contents/installation.html#gatk-and-mutect-mutect2)

Hope that works for an immediate fix,
Brad

[Stephen]

Thanks, I shortened the installation path and it seems that have installed successfully. Is there a way to test it to be sure? Maybe on a sample dataset & pipeline?

Also, I am still not sure how to get GATK registered, since the supplied command is not in my environment. Do I need to activate the Anaconda installation first? Not sure how to go about that, since I did the 'isolated' install.

gatk-register /path/to/GenomeAnalysisTK[-3.7.tar.bz2|.jar]

[Brad Chapman]

Stephen;
Glad that fixed it and you got the installation finished.

> Thanks, I shortened the installation path and it seems that have installed
> successfully. Is there a way to test it to be sure? Maybe on a sample
> dataset & pipeline?

There is a test suite with minimal datasets and details on running it are here:

http://bcbio-nextgen.readthedocs.io/en/latest/contents/testing.html#test-suite

> Also, I am still not sure how to get GATK registered, since the supplied
> command is not in my environment. Do I need to activate the Anaconda
> installation first? Not sure how to go about that, since I did the
> 'isolated' install.
> gatk-register /path/to/GenomeAnalysisTK[-3.7.tar.bz2|.jar]

You need to add the bcbio installation directory to your PATH when running it
with:

export PATH=/path_to_bcbio/bin:$PATH

http://bcbio-nextgen.readthedocs.io/en/latest/contents/installation.html#automated

That will include gatk-register and all the other tools and make them
available to bcbio. Hope this helps,
Brad