How to incorporate QIIME in an automated pipeline (after installation with Miniconda)

[Dani Haze]

Hello, I am trying to incorporate QIIME in an automated pipeline I wrote.

I have been running MacQIIME in a Mac, and it was straightforward to do, as explained here: http://www.wernerlab.org/software/macqiime/macqiime-installation#hack

Just creating a bash script (script.sh) that sources /macqiime/configs/bash_profile.txt and runs any QIIME command, as in:

#!/bin/bash

source /macqiime/configs/bash_profile.txt

print_qiime_config.py -t

... and then:
> sh script.sh

Now I want to do the exact same on a Linux machine. I installed QIIME using Miniconda, following the instructions here: http://qiime.org/install/install.html

I tried to create script.sh like this

#!/bin/bash

source activate qiime1

print_qiime_config.py -t

... and run it like:
> sh script.sh

but I get:

script.sh: 2: script.sh: source: not found
script.sh: 3: script.sh: print_qiime_config.py: not found

It only works if I do:

> source script.sh

But I need to be able to call it with "sh", cause the script will be embedded and created dinamically inside a big Perl pipeline I wrote that does a bunch of other things.

So is there a way I can reproduce the MacQIIME behavior in Linux, after QIIME installation with Miniconda? How can I incorporate QIIME in an automated pipeline the same way as in http://www.wernerlab.org/software/macqiime/macqiime-installation#hack ?

Many thanks!
Daniel

[Colin Brislawn]

Hello Daniel,

Thanks for posting! Let me see if I understand your pipeline... Does it look like this?

> big Perl pipeline

>> script.sh

>>> source activate qiime1

>>> print_qiime_config.py -t

Setting up software environments and pipelines is a little tricky, but my first thought is to try activating the qiime environment as the very first thing. Something like this.

> source activate qiime1

>> big Perl pipeline

>>> script.sh

>>>> print_qiime_config.py -t

In my own pipelines, I have had the best results activating qiime early. This will let all subsequent qiime scripts access the qiime environment, and should not interfere with your perl pipeline in any way. 

Colin

PS Have you posted that big perl pipeline anywhere? I'm always interested in seeing how other people do things!

[Dani Haze]

Hi Colin! Many thanks for your response! So a Minimal Working Example with the relevant lines in my Perl line using MacQIIME would be the following:

#!/usr/bin/perl -w
use strict;
use warnings;
my $shscript="runsh.sh";
open(SH, ">$shscript") or die "Cannot open $shscript: $!\n";
print SH '#!/bin/bash
source /macqiime/configs/bash_profile.txt
print_qiime_config.py -t
';
close(SH);
system("sh $shscript");
unlink $shscript;

It just creates a shell script on the fly that sources to MacQIIME and runs QIIME commands (not super beautiful I know). But I can't reproduce it with the QIIME1 Miniconda environment in Linux. I would like to be able to do the same with minimal changes. Changing the line:

source /macqiime/configs/bash_profile.txt

for:

source activate qiime1

does not work, and Perl cannot do:

system("source $shscript");

So what you are suggesting is doing "source activate qiime1" outside my pipeline, and then run my pipeline inside the environment, is it? That is exactly what I want to try to avoid if it's possible.

So the same way I can source the environment in a shell script with MacQIIME, can't I with Miniconda? Many thanks!

[Colin Brislawn]

Hello Dani,

Thanks for sticking with me on this. It's hard to talk about running software pipelines from within software pipelines; it's pipelines all the way down.

So what you are suggesting is doing "source activate qiime1" outside my pipeline, and then run my pipeline inside the environment, is it?

Exactly. Loading my software environment is one of the first things I do after connecting:

ssh supercomputer

source activate qiime1

cd projects/example/scripts

./perl_script1.sh

 So during while running this perl script, you have the qiime scripts 'activated' (meaning, visible in your $PATH), so any perl parts that want to call qiime scripts are able to do so. After you are done using qiime scripts, you can deactivate it with 'source deactivate'

That is exactly what I want to try to avoid if it's possible.

Why? I love keeping all my software in discrete software environments. Does some of the qiime 1.9.1 scripts conflict with your perl stuff?

Colin

[Dani Haze]

Oh I think I've replied privately before, or I don't know what happened that the message doesn't show here... hehe. Either way, not a big deal, I was just wondering if I could create a shell script wrapper that does the following, which I don't seem to beg able to do:

source activate qiime1

perl my program.pl

source deactivate

just to have everything in one single command in case someone else needs to run my pipeline... but not a problem to have to type the three commands by hand.

Many thanks!

Daniel

[Colin Brislawn]

Hello Daniel,

Yep, I got your private email. Thanks for posting it to the forms. 

You could add the activation to your .bashrc file, so that it loads automatically when you connect to the server. That saves the first step. Also, you only have to run 'source deactivate' if you are switching to a different conda environment, so that could save the final step too.

Yeah, conda was never designed to be a 'one step' solution. Before using software in a conda environment, you do have to activate it. Even worse (or better!) every user has to install their own conda environments. So if someone else is using your pipeline on a different account, they would also have to install qiime, as shown here.

I quite like this flexibility, but it's a lot more work for new users compared to 'module load qiime'

Colin