Advanced MacQIIME hacking : automated BASH script

[Dennis Jakupovic]

Dear all, 

I'm trying to automate my workflow in MacQIIME as I did before in mothur. I make use of a script that should run the whole analysis in one go, but I'm stuck at the beginning.

My simplified (test) script looks as follows:

#!/bin/bash

source /macqiime/configs/bash_profile.txt

cd ~/Desktop/HQ/1_fastq_files/

macqiime

split_libraries.fastq -i 1.fastq -m ../general/map_all.txt -o output_test/ --sample_ids 1.fastq -q 9 --barcode_type not-barcoded
#...rest of script

I manage to run the script, but a problem occurs on line 4 when the macqiime command opens the subshell in the Terminal. Here the script ends, and nothing has been processed. The rest of the script after line 4 remains untouched, and my terminal only shows that the macqiime subshell is opened and ready for use.

Does anyone recognizes this problem, or knows a fix?

I'm open for all suggestions. 

Best!
Dennis

[Jamie Morton]

Hi Dennis,

Have you tried commenting out the 3rd line?

Best,

Jamie

[Colin Brislawn]

Hello Dennis,

Try loading macqiime in the terminal first, then running your bash script with the commands. Because it's already loaded, you won't have to load it again in the script.

Colin Brislawn

[Dennis Jakupovic]

Hi Colin, 

Thanks for your reply.
I tried like you proposed, starting up the macqiime subshell first, then loading the script shown below, but I got the error "command not found".

#!/bin/bash

source /macqiime/configs/bash_profile.txt

split_libraries.fastq -i 1.fastq -m ../general/map_all.txt -o output_test/ --sample_ids 1.fastq -q 9 --barcode_type not-barcoded

Best,

Dennis

[Colin Brislawn]

Oh yeah, this is kind of tricky because you have to make sure each terminal is looking for the right things in the right places.

Try making a file called print_config.sh with these lines:

#!/bin/bash
echo Script running! Current directory is
pwd
echo Qiime config is
print_qiime_config.py -t

Then open up a new terminal, and run these commands:

EDIT: I forgot a really important step here (added in bold):

pwd
ls -alh
chmod +x print_config.sh

./print_config.sh 

macqiime

./print_config.sh

Post the full output when you have a moment. It should fail the first time you run it and work the second time. 

Sorry if this is overly specific. It's really hard to make sure that every piece is in place and is happily talking to every other piece, so all these little details add up.

Thanks!

Colin Brislawn

[Dennis Jakupovic]

Hi Colin, 

Thanks for your overly specific instructions, in these cases you cannot be specific enough I guess.

Here's the output, what do you think?

Thanks again for walking me through! 

Best, 
Dennis

>> pwd

/Users/Jakupovic/Desktop

>> ls -alh

total 80

drwx------+ 14 Jakupovic  staff   476B Feb 26 11:41 .

drwxr-xr-x+ 38 Jakupovic  staff   1.3K Feb 25 14:55 ..

-rw-r--r--@  1 Jakupovic  staff    10K Feb 26 11:41 .DS_Store

-rw-r--r--   1 Jakupovic  staff   338B Nov 20 14:44 .Rapp.history

-rw-r--r--   1 Jakupovic  staff    12K Feb 26 10:26 .Rhistory

-rw-r--r--   1 Jakupovic  staff     0B Oct  7 17:48 .localized

drwxr-xr-x   6 Jakupovic  staff   204B Feb 21 08:38 DQ

drwxr-xr-x@  6 Jakupovic  staff   204B Feb 26 10:34 FIWI

drwxr-xr-x  24 Jakupovic  staff   816B Feb 26 11:41 HQ

drwxr-xr-x   9 Jakupovic  staff   306B Feb 18 19:04 Papers_to_read

drwxrwxrwx@  4 Jakupovic  staff   136B Feb 26 11:37 To MacMini

-rwx--x--x@  1 Jakupovic  staff   103B Feb 26 11:41 print_config.sh

-rw-------@  1 Jakupovic  staff   162B Oct 14 11:57 ~$R_log.docx

-rw-------@  1 Jakupovic  staff   162B Oct 23 10:02 ~$mper analysis in run 1.docx

>> chmod +x print_config.sh

>> ./print_config.sh

Script running! Current directory is

/Users/Jakupovic/Desktop

Qiime config is

./print_config.sh: line 5: print_qiime_config.py: command not found

>> macqiime

MacQIIME version:

MacQIIME 1.9.1-20150604

Sourcing MacQIIME environment variables...

  This is the same as a normal terminal shell, except your default

  python is DIFFERENT (/macqiime/bin/python) and there are other new

 QIIME-related things in your PATH.

 Type "exit" (return) to go back to your normal shell

MacQIIME MacAir:Desktop $ ./print_config.sh

Script running! Current directory is

/Users/Jakupovic/Desktop

Qiime config is

System information

==================

        Platform:      darwin

  Python version:      2.7.10 |Anaconda 2.2.0 (x86_64)| (default, May 28 2015, 17:04:42)  [GCC 4.2.1 (Apple Inc. build 5577)]

Python executable:      /macqiime/anaconda/bin/python

QIIME default reference information

===================================

For details on what files are used as QIIME's default references, see here:

 https://github.com/biocore/qiime-default-reference/releases/tag/0.1.2

Dependency versions

===================

          QIIME library version:        1.9.1

           QIIME script version:        1.9.1

qiime-default-reference version:        0.1.2

                  NumPy version:        1.9.2

                  SciPy version:        0.15.1

                 pandas version:        0.16.1

             matplotlib version:        1.4.3

            biom-format version:        2.1.4

                   h5py version:        2.4.0 (HDF5 version: 1.8.14)

                   qcli version:        0.1.1

                   pyqi version:        0.3.2

             scikit-bio version:        0.2.3

                 PyNAST version:        1.2.2

                Emperor version:        0.9.51

                burrito version:        0.9.1

       burrito-fillings version:        0.1.1

              sortmerna version:        SortMeRNA version 2.0, 29/11/2014

              sumaclust version:        SUMACLUST Version 1.0.00

                  swarm version:        Swarm 1.2.19 [Jun  2 2015 14:40:16]

                          gdata:        Installed.

QIIME config values

===================

For definitions of these settings and to learn how to configure QIIME, see here:

 http://qiime.org/install/qiime_config.html

 http://qiime.org/tutorials/parallel_qiime.html

                     blastmat_dir:      None

      pick_otus_reference_seqs_fp:      /macqiime/anaconda/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

                         sc_queue:      all.q

      topiaryexplorer_project_dir:      None

     pynast_template_alignment_fp:      /macqiime/anaconda/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.pynast.fasta

                  cluster_jobs_fp:      start_parallel_jobs.py

pynast_template_alignment_blastdb:      None

assign_taxonomy_reference_seqs_fp:      /macqiime/anaconda/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

                     torque_queue:      friendlyq

                    jobs_to_start:      1

                       slurm_time:      None

            denoiser_min_per_core:      50

assign_taxonomy_id_to_taxonomy_fp:      /macqiime/anaconda/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt

                         temp_dir:      /tmp/

                     slurm_memory:      None

                      slurm_queue:      None

                      blastall_fp:      blastall

                 seconds_to_sleep:      60

QIIME base install test results

===============================

.........

----------------------------------------------------------------------

Ran 9 tests in 0.039s

OK

[Colin Brislawn]

Thanks for doing all these little steps for me. It worked!

Take a look at this section

>> ./print_config.sh
Script running! Current directory is
/Users/Jakupovic/Desktop
Qiime config is
./print_config.sh: line 5: print_qiime_config.py: command not found

>> macqiime
MacQIIME version:
MacQIIME 1.9.1-20150604
Sourcing MacQIIME environment variables...
  This is the same as a normal terminal shell, except your default
  python is DIFFERENT (/macqiime/bin/python) and there are other new
 QIIME-related things in your PATH.
 Type "exit" (return) to go back to your normal shell

MacQIIME MacAir:Desktop $ ./print_config.sh
Script running! Current directory is
/Users/Jakupovic/Desktop
Qiime config is

That script I had you make, print_config.sh, contains a single qiime command. That qiime comand files when you first run the script, because macqiime has not been loaded. But when you load macqiime, THEN try to run the script, it works perfectly! So the qiime commands work inside of pipeline scripts as long as you have first loaded macqiime in that terminal before running the script. 

So instead of adding 'source /macqiime/configs/bash_profile.txt' inside you scripts, type in 'macqiime' outside your script in the terminal, then run any scripts you need.

I hope that helps!

Let me know once you get this up and running.

Colin Brislawn

[Dennis Jakupovic]

Hi Colin!

Thank you so much! It's working now!

Just a question out of interest, didn't it work before because in my .bash_profile I had taken up the line 

export PATH=/Users/Jakupovic/macqiime:$PATH

and so references/relays (that line in my .bash_profile with the 'source' line in my script) became mixed up?

Thanks much again!

PS. for others who had the same problem:

       run macqiime in the terminal so it loads its own subshell

       then run your script with "sh yourscript.sh" from within the macqiime subshell.
       make sure the first line of your script is #!/bin/bash, followed by macqiime commands (.py) on the other lines.

[Colin Brislawn]

Hi Dennis,

I'm not sure why it did not work. I think sourcing it like that would work, but I guess it did not. If you keep experimenting and find a solution, let me know.

Colin